1 introduction yao-yuan chuang. 2 outline high performance computing scientific computing ...

1

Introduction

Yao-Yuan Chuang

2

Outline High Performance Computing Scientific Computing Parallel Computing UNIX, GNU, and Linux Cluster Environment Bash Linux Commands Software development process

3

High-Performance Computing The term high performance computing (HPC) refers to

the use of (parallel) supercomputers and computer clusters, that is, computing systems comprised of multiple (usually mass-produced) processors linked together in a single system with commercially available interconnects.

This is in contrast to mainframe computers, which are generally monolithic in nature.

Because of their flexibility, power, and relatively low cost, HPC systems increasingly dominate the world of supercomputing. Usually, computer systems in or above the teraflop-region are counted as HPC-computers.

4

Scientific Computing “Scientific computing (or computational science) is the

field of study concerned with constructing mathematical models and numerical solution techniques and using computers to analyze and solve scientific and engineering problems.

In practical use, it is typically the application of computer simulation and other forms of computation to problems in various scientific disciplines.

Scientists and engineers develop computer programs, application software, that model systems being studied and run these programs with various sets of input parameters. Typically, these models require massive amounts of calculations (usually floating-point) and are often executed on supercomputers or distributed computing platforms.

5

Problem Domains for Computational Science/Scientific Computing

Numerical simulations Numerical simulations have different objectives depending on the

nature of the task being simulated: Reconstruct and understand known events (e.g., earthquake, tsunamis

and other natural disasters). Predict future or unobserved situations (e.g., weather, sub-atomic

particle behavior).

Model fitting and data analysis Appropriately tune models or solve equations to reflect observations,

subject to model constraints (e.g. oil exploration geophysics, computational linguistics)

Use graph theory to model networks, especially those connecting individuals, organizations, and websites.

Optimization Optimize known scenarios (e.g., technical and manufacturing

processes, front end engineering).

6

Supercomputer A supercomputer is a computer

that leads the world in terms of processing capacity, particularly speed of calculation, at the time of its introduction. The term "Super Computing" was first used by New York World newspaper in 1920 to refer to large custom-built tabulators IBM made for

Columbia University.

7

IBM Roadrunner, LLNL, USA, 1.026 PFLOPS

8

Moore’s Law Moore's Law is the

empirical observation made in 1965 that the number of transistors on an integrated circuit for minimum component cost doubles every 24 months.

The most popular

formulation is of the doubling of the number of transistors on integrated circuits every 18 months.

9

FLOPS In computing, FLOPS (or flops) is an acronym meaning FLoa

ting point Operations Per Second. This is used as a measure of a computer's performance, especially in fields of scientific calculations that make heavy use of floating point calculations; similar to million instructions per second (MIPS).

The standard SI prefixes can be used for this purpose, resulting in such units as megaFLOPS (MFLOPS, 106), gigaFLOPS (GFLOPS, 109), teraFLOPS (TFLOPS, 1012), petaFLOPS (PFLOPS, 1015) and exaFLOPS (EFLOPS, 1018).

As of 2008 the fastest supercomputer's performance tops out at one petaflop.

10

www.top500.org

11

www.linuxhpc.org

12

www.winhpc.org

13

Intel’s 80 Core Chip

Performance numbers*at 4.27 Ghz:peak performance:* 1.37 SP TFLOPSExplicit PDE solver:* 1 SP TFLOPSmatrix multiplication:•0.51 SP TFLOPS•2 independent FMAC units – 2 Single Precision FLOPS per cycle

14

Compiler A programming language is an

artificial language that can be used to control the behavior of a machine, particularly a computer.

A compiler is a computer program (or set of programs) that translates text written in a computer language (the source language) into another computer language (the target language). The original sequence is usually called the source code and the output called object code. Commonly the output has a form suitable for processing by other programs (e.g., a linker), but it may be a human-readable text file.

15

Debug and Performance Analysis Debugging is a methodical process of finding and reducing

the number of bugs, or defects, in a computer program or a piece of electronic hardware thus making it behave as expected.

In software engineering, performance analysis (a field of dynamic program analysis) is the investigation of a program's behavior using information gathered as the program runs, as opposed to static code analysis. The usual goal of performance analysis is to determine which parts of a program to optimize for speed or memory usage.

A profiler is a performance analysis tool that measures the behavior of a program as it runs, particularly the frequency and duration of function calls. The output is a stream of recorded events (a trace) or a statistical summary of the events observed (a profile).

16

Optimization In computing, optimization is the process of

modifying a system to make some aspect of it work more efficiently or use less resources. For instance, a computer program may be optimized so that it executes more rapidly, or is capable of operating within a reduced amount of memory storage, or draws less battery power in a portable computer. The system may be a single computer program, a collection of computers or even an entire network such as the Internet.

(http://en.wikipedia/org/wiki/Optimization_%28computer_science%29)

17

Parallel Computing Parallel computing is the simultaneous execution of the

same task (split up and specially adapted) on multiple processors in order to obtain results faster.

The idea is based on the fact that the process of solving a problem usually can be divided into smaller tasks, which may be carried out simultaneously with some coordination.

Flynn's taxonomy whether all processors execute the same instructions at the

same time (single instruction/multiple data -- SIMD) each processor executes different instructions (multiple

instruction/multiple data -- MIMD).

(http://en.wikipedia/org/wiki/Parallel_computing)

18

Terminology in parallel computing Efficiency

is the execution time using a single processor divided by the quantity of the execution time using a multiprocessor and the number of procesors.

Parallel Overhead the extra work associated with parallel version compared to its sequential code,

mostly the extra CPU time and memory space requirements from synchronization, data communications, parallel environment creation and cancellation, etc.

Synchronization the coordination of simultaneous tasks to ensure correctness and avoid unexpe

cted race conditions. Speedup

also called parallel speedup, which is defined as wall-clock time of best serial execution divided by wall-clock time of parallel execution. Amdahl's law can be used to give a maximum speedup factor.

Scalability a parallel system's ability to gain proportionate increase in parallel speedup wi

th the addition of more processors. Also, see this Parallel Computing Glossary Task

a logically high level, discrete, independent section of computational work. A task is typically executed by a processor as a program

19

Amdahl's law Amdahl's law, named after computer architect Gene Amdahl, is

used to find the maximum expected improvement to an overall system when only part of the system is improved. It is often used in parallel computing to predict the theoretical maximum speedup using multiple processors.

Pk is a percentage of the instructions that can be improved (or slowed),

Sk is the speed-up multiplier (where 1 is no speed-up and no slowing),

k represents a label for each different percentage and speed-up, and

n is the number of different speed-up/slow-downs resulting from the system change.

n

kk

k

SP

0

1

20

Parallelism One of the simplest methods used to accomplish

increased parallelism is to begin the first steps of instruction fetching and decoding before the prior instruction finishes executing. This is the simplest form of a technique known as instruction pipelining, and is utilized in almost all modern general-purpose CPUs. Pipelining allows more than one instruction to be executed at any given time by breaking down the execution pathway into discrete stages.

Another strategy of achieving performance is to execute multiple programs or threads in parallel. This area of research is known as parallel computing. In Flynn's taxonomy, this strategy is known as Multiple Instructions-Multiple Data or MIMD.

A less common but increasingly important paradigm of CPUs (and indeed, computing in general) deals with data parallelism. A vector processor, or array processor, is a CPU design that is able to run mathematical operations on multiple data elements simultaneously.

21

Parallelism In computing, multitasking is a method by which

multiple tasks, also known as processes, share common processing resources such as a CPU.

Multiprocessing is a generic term for the use of two or more central processing units (CPUs) within a single computer system. It also refers to the ability of a system to support more than one processor and/or the ability to allocate tasks between them

A multi-core microprocessor is one that combines two or more independent processors into a single package, often a single integrated circuit (IC). A dual-core device contains two independent microprocessors. In general, multi-core microprocessors allow a computing device to exhibit some form of thread-level parallelism (TLP) without including multiple microprocessors in separate physical packages. This form of TLP is often known as chip-level multiprocessing.

22

Computer Cluster

A computer cluster is a group of tightly coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. The components of a cluster are commonly, but not always, connected to each other through fast local area networks. Clusters are usually deployed to improve performance and/or availability over that provided by a single computer, while typically being much more cost-effective than single computers of comparable speed or availability.

23

High-Availability (HA) Clusters High-availability clusters (a.k.a Failover

clusters) are implemented primarily for the purpose of improving the availability of services which the cluster provides. They operate by having redundant nodes, which are then used to provide service when system components fail. The most common size for an HA cluster is two nodes, which is the minimum requirement to provide redundancy. HA cluster implementations attempt to manage the redundancy inherent in a cluster to eliminate single points of failure. There are many commercial implementations of High-Availability clusters for many operating systems. The Linux-HA project is one commonly used free software HA package for the Linux OS.

24

Load-balancing Clusters Load-balancing clusters operate by having all workload

come through one or more load-balancing front ends, which then distribute it to a collection of back end servers. Although they are primarily implemented for improved performance, they commonly include high-availability features as well. Such a cluster of computers is sometimes referred to as a server farm.

There are many commercial load balancers available including Platform LSF HPC, Sun Grid Engine, Moab Cluster Suite and Maui Cluster Scheduler. The Linux Virtual Server project provides one commonly used free software package for the Linux OS.

For example, web farm of Google and Yahoo

25

High-performance Clusters High-performance clusters are implemented primarily to provi

de increased performance by splitting a computational task across many different nodes in the cluster, and are most commonly used in scientific computing.

One of the most popular HPC implementations is a cluster with nodes running Linux as the OS and free software to implement the parallelism. This configuration is often referred to as a Beowulf cluster.

HPCs are optimized for workloads which require jobs or processes happening on the separate cluster computer nodes to communicate actively during the computation. These include computations where intermediate results from one node's calculations will affect future calculations on other nodes.

26

Beowulf (computing)

Beowulf is a design for high-performance parallel computing clusters on inexpensive personal computer hardware. Originally developed by Thomas L. Sterling and Donald Becker at NASA, Beowulf systems are now deployed worldwide, chiefly in support of scientific computing.

A Beowulf cluster is a group of usually identical PC computers networked into a small TCP/IP LAN, and have libraries and programs installed which allow processing to be shared among them.

There is no particular piece of software that defines a cluster as a Beowulf. Commonly used parallel processing libraries include MPI (Message Passing Interface) and PVM (Parallel Virtual Machine). Both of these permit the programmer to divide a task among a group of networked computers, and recollect the results of processing.

27

OpenMosix openMosix is a free cluster managemen

t system that provides single-system image (SSI) capabilities, e.g. automatic work distribution among nodes. It allows program processes (not threads) to migrate to machines in the node's network that would be able to run that process faster (process migration). It is particularly useful for running parallel and intensive input/output (I/O) applications. It is released as a Linux kernel patch, but is also available on specialized LiveCDs and as a Gentoo Linux kernel choice.

openMosix is currently considered stable on 2.4 kernel for the x86 architecture.

28

Pthread POSIX Threads is a POSIX standard for threads. The standard def

ines an API for creating and manipulating threads.

POSIX or "Portable Operating System Interface for uniX" is the collective name of a family of related standards specified by the IEEE to define the application programming interface (API) for software compatible with variants of the Unix operating system.

A thread in computer science is short for a thread of execution. Threads are a way for a program to split itself into two or more simultaneously (or pseudo-simultaneously) running tasks. Threads and processes differ from one operating system to another, but in general, the way that a thread is created and shares its resources is different from the way a process does.

29

Message Passing Interface The Message Passing Interface (MPI) is a language-independent

computer communications descriptive application programmer interface (API) with defined semantics and flexible interpretations; it does not define the protocol by which these operations are to be performed in the sense of sockets for TCP/IP or other layer-4 and below models in the ISO/OSI Reference Model.

It is consequently a layer-5+ type set of interfaces, although implementations can cover most layers of the reference model, with sockets+TCP/IP as a common transport used inside the implementation. MPI's goals are high performance, scalability, and portability.

Productivity of the interface for programmers is not one of the key goals of MPI, and MPI is generally considered to be low-level. It expresses parallelism explicitly rather than implicitly. MPI is considered successful in achieving high performance and high portability, but is often criticized for its low-level qualities.

30

TCP/IP Application Transport layer (TCP) Network layer(IP) Data link Physical layer

31

OpenMP The OpenMP (Open Multi-Processing)

application programming interface (API) supports multi-platform shared memory multiprocessing programming in C/C++ and Fortran on many architectures, including Unix and Microsoft Windows platforms. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior.

Jointly defined by a group of major computer hardware and software vendors, OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications for platforms ranging from the desktop to the supercomputer.

(http://en.wikipedia.org/wiki/Openmp)

32

Distributed Shared Memory Distributed Shared Memory (DSM), in computer science,

refers to a wide class of software and hardware implementations, in which each node of a cluster has access to a large shared memory in addition to each node's limited non-shared private memory.

Software DSM systems can be implemented within an operating system, or as a programming library. Software DSM systems implemented in the operating system can be thought of as extensions of the underlying virtual memory architecture. Such systems are transparent to the developer; which means that the underlying distributed memory is completely hidden from the users. In contrast, Software DSM systems implemented at the library or language level are not transparent and developers usually have to program differently. However, these systems offer a more portable approach to DSM system implementation.

33

Distributed Computing Distributed computing is a method of computer

processing in which different parts of a program run simultaneously on two or more computers that are communicating with each other over a network.

Distributed computing is a type of parallel processing. But the latter term is most commonly used to refer to processing in which different parts of a program run simultaneously on two or more processors that are part of the same computer.

While both types of processing require that a program be parallelized -- divided into sections that can run simultaneously, distributed computing also requires that the division of the program take into account the different environments on which the different sections of the program will be running. For example, two computers are likely to have different file systems and different hardware components.

(http://en.wikipedia/org/wiki/Distributed_computing)

34

BOINC The Berkeley Open Infrastructure for Network Computing (B

OINC) is a middleware system for volunteer computing, originally developed to support SETI@home, but intended to be useful for other applications as well.

Currently BOINC is being developed by a team based at the University of California, Berkeley led by David Anderson, who also leads SETI@home. As a "quasi-supercomputing" platform BOINC has over 435,000 active computers (hosts) worldwide processing on average 500 TFLOPS as of January 30, 2007

boinc.berkeley.edu

35

Grid Computing Grid computing or grid clusters are a technology closely

related to cluster computing. The key differences between grids and traditional clusters are that grids connect collections of computers which do not fully trust each other, and hence operate more like a computing utility than like a single computer. In addition, grids typically support more heterogeneous collections than are commonly supported in clusters.

Grid computing is optimized for workloads which consist of many independent jobs or packets of work, which do not have to share data between the jobs during the computation process. Grids serve to manage the allocation of jobs to computers which will perform the work independently of the rest of the grid cluster. Resources such as storage may be shared by all the nodes, but intermediate results of one job do not affect other jobs in progress on other nodes of the grid.

36

Topics in Parallel Computing General High-performance computing Parallelism Data parallelism • Task parallelism Theory Speedup • Amdahl's law • Flynn's Taxonomy • Cost efficiency •

Gustafson's Law • Karp-Flatt Metric Elements Process • Thread • Fiber • Parallel Random Access Machine Multiprocessing Multitasking • Memory coherency • Cache coherency •

Barrier • Synchronization • Distributed computing • Grid computing Programming Programming model • Implicit parallelism •

Explicit parallelism Hardware Computer cluster • Beowulf • Symmetric multiprocessing •

Non-Uniform Memory Access • Cache only memory architecture • Asymmetric multiprocessing • Simultaneous multithreading • Shared memory • Distributed memory • Massively parallel processing • Superscalar processing • Vector processing • Supercomputer

Software Distributed shared memory • Application checkpointing APIs Pthreads • OpenMP • Message Passing Interface (MPI) Problems Embarrassingly parallel • Grand Challenge

37

UNIX Unix (officially trademarked as UNIX®) is a computer operating system ori

ginally developed in the 1960s and 1970s by a group of AT&T employees at Bell Labs including Ken Thompson, Dennis Ritchie and Douglas McIlroy. Today's Unix systems are split into various branches, developed over time by AT&T as well as various commercial vendors and non-profit organizations.

The present owner of the trademark UNIX® is The Open Group, an industry standards consortium. Only systems fully compliant with and certified to the Single UNIX Specification qualify as "UNIX®" (others are called "Unix system-like" or "Unix-like").

During the late 1970s and early 1980s, Unix's influence in academic circles led to large-scale adoption (particularly of the BSD variant, originating from the University of California, Berkeley) of Unix by commercial startups, the most notable of which is Sun Microsystems. Today, in addition to certified Unix systems, Unix-like operating systems such as Linux, Mac OS X and BSD derivatives are commonly encountered.

38

UNIX

(http://en.wikipedia/org/wiki/Unix)

39

GNU The GNU project was publicly announced by Richard Stallman.

GNU was to be a complete Unix-like operating system composed entirely of free software. Software development work began in January 1984. By the beginning of the 1990s, the project had produced or collected most of the necessary components of this system, including libraries, compilers, text editors, and a Unix shell.

Thus the GNU mid-level portions of the operating system were almost complete. The upper level could be supplied by the X Window System, but the lower level, which consisted of a kernel, device drivers, and daemons, was still mostly lacking. In 1990, the GNU project began developing the GNU Hurd kernel, based on the Mach microkernel.

(http://en.wikipedia/org/wiki/GNU)

40

Linux Linus Torvalds, creator of the Linux kernel.

In 1991, work on the Linux kernel began by Linus Torvalds while attending the University of Helsinki. Torvalds originally created the Linux kernel as a non-commercial replacement for the Minix kernel; he later changed his original non-free license to the GPLv2, which differed primarily in that it also allowed for commercial redistribution.

Although dependent on the Minix userspace at first, work from both Linux kernel developers and the GNU project allowed Linux to work with GNU components. Thus Linux filled the last major gap in running a complete, fully functional operating system built from free software.

(http://en.wikipedia/org/wiki/Linux)

41

Kernel In computing, the kernel is the central component of most

computer operating systems (OSs). Its responsibilities include managing the system's resources and the communication between hardware and software components.

As a basic component of an operating system, a kernel provides the lowest-level abstraction layer for the resources (especially memory, processors and I/O devices) that applications must control to perform their function. It typically makes these facilities available to application processes through inter-process communication mechanisms and system calls.

While monolithic kernels will try to achieve these goals by executing all the code in the same address space to increase the performance of the system, microkernels run most of their services in user space, aiming to improve maintainability and modularity of the codebase.

42

Linux Distribution A Linux distribution, often simply distribution or dis

tro, is a member of the Linux family of Unix-like operating systems comprised of the Linux kernel, the non-kernel parts of the GNU operating system, and assorted other software. Linux distributions take a variety of forms, from fully-featured desktop and server operating systems to minimal environments (typically for use in embedded systems, or for booting from a floppy).

CentOS • Debian • Fedora • Gentoo • Knoppix • Mandriva Linux • Red Hat Enterprise Linux • Slackware • SUSE Linux • Ubuntu • more… •

43

Unix Shell A Unix shell, also called "the command line", provides the tr

aditional user interface for the Unix operating system and for Unix-like systems.

bash – Bourne Again SHell, (mostly) sh-compatible and csh-compatible, standard shell on Linux systems and Mac OS X.

csh – C shell. Written by Bill Joy for BSD systems. ksh – Korn shell, standard shell on many proprietary Unix sys

tems, powerful successor to the Unix Bourne shell (sh), written by David Korn,

rc – originally written for Plan 9. sh – Bourne shell, only shell present on all UNIX and Unix-like

systems, written by Stephen Bourne for Version 7 Unix. tcsh – TENEX C shell, standard shell on BSD systems. zsh – Z shell.

(http://en.wikipedia/org/wiki/Unix_shell)

44

Bash Bash is a Unix shell written for the GNU Project. The name of the a

ctual executable is bash. Its name is an acronym for Bourne-again shell, a pun ('Bourne again' / 'born again') on the name of the Bourne shell (sh), an early and important Unix shell written by Stephen Bourne and distributed with Version 7 Unix circa 1978. Bash was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer.

Bash is the default shell on most Linux systems as well as on Mac OS X and it can be run on most Unix-like operating systems. It has also been ported to Microsoft Windows within the Cygwin POSIX emulation environment for Windows and to MS-DOS by the DJGPP project. Released under the GNU General Public License, Bash is free software.

(http://en.wikipedia/org/wiki/Bash)

45

Math Cluster

46

Math Cluster 控制節點

對外 IP ： 140.127.223.11 (eth1) 對內 IP ： 192.168.0.254 (eth0)

計算節點 192.168.0.1 ~7 (node1 ~ node7)

帳號 hpc001 …

47

Remote Login:SSH Client

Don’t forget change the password after first time login

48

Kernel Specific CommandsKernel specific date – Print or set the system date and/or time dmesg – Print the kernel message buffer ipcrm – Remove a message queue, semaphore

set or shared memory id ipcs – Provide information on IPC facilities uname – Print assorted system statistics

(http://en.wikipedia/org/wiki/Linux_commands)

49

General User Commands dd – Convert and copy a file (Disk Dump) dirname – Strip non-directory suffixes from a path echo – Print to standard output env – Show environment variables; run a program wit

h altered environment variables file (or stat) – Determine the type of a file nohup – Run a command with immunity to hangups o

utputting to non-tty sh – The Bourne shell, the standard Unix shell uptime – Print how long the system has been running


50

Processes and tasks management anacron – Periodic command scheduler at – Single-time command scheduler chroot – Change the system root directory for all child processes cron – Periodic command scheduler crontab – Crontab file editor daemonic – Interface to daemon init scripts htop – Interactive ncurses-based process viewer that allows scrol

ling to see all processes and their full command lines kill – Send a signal to process, or terminate a process (by PID) killall – Terminate all processes (in GNU/Linux, it's kill by name)


51

Processes and tasks management nice – Alter priorities for processes pgrep – Find PIDs of processes by name pidof – GNU/Linux equivalent of pgrep pkill – Send a signal to process, or terminate a process (by name). Equi

valent to Linux killall ps – Report process status renice – Alter the priorities of an already running process sleep – Delay for specified time time – Time a command timex – Time process shell execution, measure process data and syste

m activity top – Produce a dynamic list of all resident processes wait – Wait for the specified process


52

User management and support chsh – Change user shell finger – Get details about user id – Print real/effective UIDs/GIDs last – show listing of last logged in users lastlog – show last log in information for users locale – Get locale specific information localedef – Compile locale definitions logname – Print user's login name man – Manual browser mesg – Control write access to your terminal passwd – Change user password


53

User management and support su – Start a new process (defaults to shell) as a different

user (defaults to root) sudo – execute a command as a different user. users – Show who is logged on (only users names) w – Show logged-in users and their current tasks whatis – command description from whatis database whereis – locates the command's binary and manual pa

ges associated with it which (Unix) – locates where a command is executed fr

om who – Show who is logged on (with some details) write – Send a message to another user


54

Filesystem Utilities info – The GNU alternative to man man – The standard unix documentation system

chattr – Change file attributes on a Linux second extended file system

chgrp – Change the group of a file or directory chmod – Change the permissions of a file or directory chown – Change the owner of a file or directory cd – Change to another directory location cp – Copy a file or directory to another location

55

Filesystem Utilities df – Report disk space dircmp – Compare contents of files between two directories du – Calculate used disk space fdupes – Find or remove duplicate files within a directory find – Search for files through a directory hierarchy fsck – Filesystem check ln – Link one file/directory to another ls – List directory contents lsattr – List file attributes on a Linux second extended file system lsof – list open files

56

Filesystem Utilities mkdir – Make a directory mkfifo – Make a named pipe mount – Mount a filesystem mv – Move or rename a file or directory pwd – Print the current working directory rm – Delete a file or directory tree readlink – Display value of a symbolic link, or display canonical p

ath for a file rmdir – Delete an empty directory touch – Create a new file or update its modification time tree – Print a depth-indented tree of a given directory unlink – System call to remove a file or directory

57

Archivers and compression afio – Compatible superset of cpio with added functionality ar – Maintain, modify, and extract from archives. Now largely obs

oleted by tar bzip2 – Block-sorting file compressor compress – Traditional compressor using the LZW algorithm cpio – A traditional archiving tool/format gzip – The gzip file compressor p7zip – 7zip for unix/linux pack, pcat, unpack – included in old versions of ATT Unix. Uses

Huffman coding, obsoleted by compress. pax – POSIX archive tool that handles multiple formats. tar – Tape ARchiver, concatenates files uncompress – Uncompresses files compressed with compress. zcat – Prints files to stdout from gzip archives without unpacking

them to separate file(s)

58

Text Processing awk – A pattern scanning and processing language banner – Creates ascii art version of an input string for printing large banners cat – Concatenate files to standard output cksum – Print the CRC checksum and bytecount of a file (see also MD5) cmp – Compare two files byte for byte cut – Remove sections from each line of a file or standard input diff – Compare two text files line by line egrep – Extended pattern matching (synonym for "grep -E") fgrep – Simplified pattern matching (synonym for "grep -F") fold – Wrap each input line to fit within the given width grep – Print lines matching a pattern head – Output the first parts of a file iconv – Convert the encoding of the specified files join – Join lines of two files on a common field less – Improved more-like text pager

59

Text Processing more – Pager nroff – Fixed-width (non-typesetter) version of the standard Unix types

etting system patch – Change files based on a patch file sed – Stream EDitor sort – Sort lines of text files split – Split a file into pieces tail – Output the tail end of files tee – Read from standard input, write to standard output and files uudecode – Decodes a binary file that was used for transmission using

electronic mail uuencode – Encodes a binary file for transmission using electronic mail wc – Word/line/byte count

60

Text Editors GNU Emacs – Freely programmable full-screen text editor and gene

ral computing environment (using builtin Elisp, a simple dialect of the Lisp programming language)

Joe – a screen-oriented text editor using a Wordstar-style command set

Jove – a screen-oriented text editor using an Emacs-style command set

pico – PIne's message COmposition editor (simple, easy to use screen editor)

vi – "Visual" (screen-oriented) text editor (originally ex in screen-oriented "visual" mode)

vim – Vi IMproved, portable vi-compatible editor with multiple buffers, screen splitting, syntax highlighting and a lot of other features not found in standard ex/vi

XEmacs – Popular version of emacs that is derived from GNU Emacs

61

Communication ftp, sftp – File transfer protocol, secure FTP NFS – Network filesystem OpenVPN – virtual private (encrypting) networking software Postfix — mail transfer agent rsh, SSH, telnet – Remote login Samba – SMB and CIFS client and server for UNIX Sendmail – popular E-Mail transport software talk – Talk to another logged-in user uustat – a Basic Networking Utilities (BNU) command that displa

ys status information about several types of BNU operations uux – Remote command execution over UUCP

62

Network monitoring and security Ethereal and tethereal – a feature rich protocol analyzer (now called Wireshark,

see below) John the Ripper – password cracking software Nessus – a comprehensive open-source network vulnerability scanning progra

m Netstat – displays a list of the active network connections the computer Nmap – free port scanning software SAINT – System Administrator’s Integrated Network Tool – Network Vulnerabi

lity Scanner. SATAN – the Security Administrator Tool for Analyzing Networks – a testing and

reporting tool that collects information about networked hosts Snoop – Solaris packet sniffer Snort – an open source network intrusion detection system tcpdump – a computer network debugging tool that intercepts and displays TC

P/IP packets being transmitted or received Wireshark – a protocol analyzer, or "packet sniffer", similar to tcpdump, that ad

ds a GUI frontend, and more sorting and filtering options (formerly named Ethereal)

63

Programming Tools bash – Bourne Again SHell, (mostly) sh-compatible and csh-compatible,

standard shell on Linux systems and Mac OS X. csh – C shell. Written by Bill Joy for BSD systems. ksh – Korn shell, standard shell on many proprietary Unix systems, powe

rful successor to the Unix Bourne shell (sh), written by David Korn, rc – originally written for Plan 9. sh – Bourne shell, only shell present on all UNIX and Unix-like systems, w

ritten by Stephen Bourne for Version 7 Unix. tcsh – TENEX C shell, standard shell on BSD systems. zsh – Z shell. awk – Standard Unix pattern scanning and text processing tool. perl – Perl scripting language. PHP – PHP scripting language. Python – Python scripting language.

64

Compilers as – GNU assembler tool. c99 – C programming language. cc – C compiler. dbx – (System V and BSD) Symbolic debugger. f77 – Fortran 77 compiler. gcc – GNU Compiler Collection C frontend (also known as GNU C Compiler) gdb – GNU symbolic debugger. ld – Program linker. lex – Lexical scanner generator. ltrace – (Linux) Trace dynamic library calls in the address space of the watched proces

s. m4 – Macro language. make – Automate builds. nm – List symbols from object files. size – return the size of the sections of an ELF file. strace – (Linux) or truss (Solaris) Trace system calls with their arguments and signals.

Useful debugging tool, but does not trace calls outside the kernel, in the address space of the process(es) being watched.

65

Desktops/Graphical User Interfaces CDE – Common Desktop Environment, most commonly found on

proprietary UNIX systems Enlightenment – an open source window manager for the X Wind

ow System FVWM and its variant FVWM95, which has been modified to beha

ve like Windows 95 Also FVWM-Crystal that aims to be eye candy GNOME – GNU Network Object Model Environment IceWM – ICE Window Manager JWM – Joe's Window Manager KDE – K Desktop Environment XFce – a desktop environment for Unix and other Unix-like platfo

rms

66

Package Management apt – Front-end for dpkg or rpm debconf – Debian package configuration management system dpkg – The Debian package manager drakconf – Front-end configuration utility for Mandriva Linux emerge – A frontend to portage pacman – A package manager used primarily by Arch Linux portage – The Gentoo Linux package manager rpm – Originally the package manager for Red Hat Linux, now used

by several distributions including Mandriva Linux Synaptic – GTK+ frontend for the apt package manager. Primarily u

sed by Ubuntu Linux, Debian Sarge, and other Debian-based systems; but usable on any system using apt.

urpmi – Front-end to rpm, used by Mandriva Linux YaST - System management utility mainly used by SuSE yum - Front-end for rpm, used by Fedora

67

Web Browsers Dillo – Extremely light-weight web browser ELinks – Enhanced links Epiphany – Light-weight GNOME web browser Galeon – Light-weight old GNOME web browser Konqueror – KDE web browser Links – Console based web browser lynx – Console based web browser Mozilla Application Suite – Graphical cross platform web browser

& email client Mozilla Firefox – Extensible Web browser Opera – Web browser and e-mail client (Proprietary Software) w3m – Console based web browser

68

Desktop Publishing groff – Traditional typesetting system LaTeX – Popular TeX macro package for higher-level typ

esetting lp – Print a file (on a line printer) Passepartout – Desktop publishing program pr – Convert text files for printing Scribus – Desktop publishing program TeX – Macro-based typesetting system troff – The original and standard Unix typesetting system

69

Databases DB2 Firebird MySQL Oracle PostgreSQL Progress Software SQLite Sybase

70

Math Tools maxima – Symbol manipulation program. octave – Numerical computing language (mostly com

patible with Matlab) and environment. R – Statistical programming language. units – Unit conversion program. bc – An arbitrary precision calculator language with sy

ntax similar to the C programming language. cal – Displays a calendar dc – Reverse-Polish desk calculator which supports un

limited precision arithmetic fortune – Fortune cookie program that prints a rando

m quote

72

Software development process A software development process is a structure imposed

on the development of a software product. Synonyms include software lifecycle and software process. There are several models for such processes, each describing approaches to a variety of tasks or activities that take place during the process. Software Elements Analysis Specification Software architecture Implementation (or coding) Testing Documentation Software Training and Support Maintenance

73

Software Elements Analysis The most important task in creating a software

product is extracting the requirements.

Customers typically know what they want, but not what software should do, while incomplete, ambiguous or contradictory requirements are recognized by skilled and experienced software engineers.

Frequently demonstrating live code may help reduce the risk that the requirements are incorrect.

74

Specification Specification is the task of precisely describing the software

to be written, possibly in a rigorous way.

In practice, most successful specifications are written to understand and fine-tune applications that were already well-developed, although safety-critical software systems are often carefully specified prior to application development.

Specifications are most important for external interfaces that must remain stable.

75

Software Architecture The architecture of a software system refers to an abstract

representation of that system. Architecture is concerned with making sure the software system will meet the requirements of the product, as well as ensuring that future requirements can be addressed. The architecture step also addresses interfaces between the software system and other software products, as well as the underlying hardware

or the host operating system.

76

Implementation and Testing Implementation (or coding): Reducing a design to code may

be the most obvious part of the software engineering job, but it is not necessarily the largest portion.

Testing: Testing of parts of software, especially where code by two different engineers must work together, falls to the software engineer.

77

Documentation and Training An important (and often overlooked) task is documenting the int

ernal design of software for the purpose of future maintenance and enhancement. Documentation is most important for external interfaces.

Common types of computer hardware/software documentation include online help, FAQs, HowTos, and user guides. The term RTFM is often used in regard to such documentation, especially to computer hardware and software user guides.

RTFM is an initialism for the statement "Read The Fucking Manual." This instruction is sometimes given in response to a question when the person being asked believes that the question could be easily answered by reading the relevant "manual" or instructions.

Some people prefer the backronym "Read The Fine Manual." Alternatively, the "F" can be dropped entirely and the initialism rendered as "RTM" (Read The Manual), or the more polite "RTMP" (Read The Manual Please).

78

Software Training and Support A large percentage of software projects fail because the

developers fail to realize that it doesn't matter how much time and planning a development team puts into creating software if nobody in an organization ends up using it. People are occasionally resistant to change and avoid venturing into an unfamiliar area, so as a part of the deployment phase, its very important to have training classes for the most enthusiastic software users (build excitement and confidence), shifting the training towards the neutral users intermixed with the avid supporters, and finally incorporate the rest of the organization into adopting the new software. Users will have lots of questions and software problems which leads to the next phase of software.

79

Maintenance Maintaining and enhancing software to cope with newly discover

ed problems or new requirements can take far more time than the initial development of the software. Not only may it be necessary to add code that does not fit the original design but just determining how software works at some point after it is completed may require significant effort by a software engineer. About of al⅔l software engineering work is maintenance, but this statistic can be misleading. A small part of that is fixing bugs. Most maintenance is extending systems to do new things, which in many ways can be considered new work. In comparison, about of all civil en⅔gineering, architecture, and construction work is maintenance in a similar way.

80

Software Developing Models Waterfall model Spiral model Model driven development User experience Top-down and bottom-up design Chaos model Evolutionary prototyping Prototyping V model Extreme Programming Hysterical raisins

81

Waterfall Model The waterfall model is a

sequential software development model (a process for the creation of software) in which development is seen as flowing steadily downwards (like a waterfall) through the phases of requirements analysis, design, implementation, testing (validation), integration, and maintenance. The origin of the term "waterfall" is often cited to be an article published in 1970 by W. W. Royce

82

Spiral Model The new system requirements are defined in as much detail

as possible. This usually involves interviewing a number of users representing all the external or internal users and other aspects of the existing system.

A preliminary design is created for the new system.

A first prototype of the new system is constructed from the preliminary design. This is usually a scaled-down system, and represents an approximation of the characteristics of the final product.

A second prototype is evolved by a fourfold procedure: (1) evaluating the first prototype in terms of its strengths, weaknesses, and risks; (2) defining the requirements of the second prototype; (3) planning and designing the second prototype; (4) constructing and testing the second prototype.

83

Spiral Model At the customer's option, the entire project can be aborted

if the risk is deemed too great. Risk factors might involve development cost overruns, operating-cost miscalculation, or any other factor that could, in the customer's judgment, result in a less-than-satisfactory final product.

The existing prototype is evaluated in the same manner as was the previous prototype, and, if necessary, another prototype is developed from it according to the fourfold procedure outlined above.

The preceding steps are iterated until the customer is satisfied that the refined prototype represents the final product desired.

The final system is constructed, based on the refined prototype.

The final system is thoroughly evaluated and tested. Routine maintenance is carried out on a continuing basis to prevent large-scale failures and to minimize downtime.

84

Top-Down and Bottom-up Top-down and bottom-up are strategies of information processing and

knowledge ordering, mostly involving software, and by extension other humanistic and scientific system theories (see systemics).

In the top-down model an overview of the system is formulated, without going into detail for any part of it. Each part of the system is then refined by designing it in more detail. Each new part may then be refined again, defining it in yet more detail until the entire specification is detailed enough to validate the model. The top-down model is often designed with the assistance of "dark boxes" that make it easier to bring to fulfillment but insufficient and irrelevant in understanding the elementary mechanisms.

In bottom-up design, first the individual parts of the system are specified in great detail. The parts are then linked together to form larger components, which are in turn linked until a complete system is formed. This strategy often resembles a "seed" model, whereby the beginnings are small, but eventually grow in complexity and completeness.

85

Software prototyping Identify basic requirements

Determine basic requirements including the input and output information desired. Details, such as security, can typically be ignored.

Develop Initial Prototype The initial prototype is developed that includes only user

interfaces. Review

The customers, including end-users, examine the prototype and provide feedback on additions or changes.

Revise and Enhancing the Prototype Using the feedback both the specifications and the prototype

can be improved. Negotiation about what is within the scope of the contract/product may be necessary. If changes are introduced then a repeat of steps #3 ands #4 may be needed.

86

Extreme Programming Extreme Programming (XP) is a

software engineering methodology, the most prominent of several agile software development methodologies. Like other agile methodologies, Extreme Programming differs from traditional methodologies primarily in placing a higher value on adaptability than on predictability. Proponents of XP regard ongoing changes to requirements as a natural, inescapable and desirable aspect of software development projects; they believe that being able to adapt to changing requirements at any point during the project life is a more realistic and better approach than attempting to define all requirements at the beginning of a project and then expending effort to control changes to the requirements.

87

Software Architecture Software architecture is commonly organized in views,

which are analogous to the different types of blueprints made in building architecture. Some possible views (actually, viewpoints in the 1471 ontology) are: Functional/logic view Code view Development/structural view Concurrency/process/thread view Physical/deployment view User action/feedback view

Several languages for describing software architectures have been devised, but no consensus has yet been reached on which symbol-set and view-system should be adopted. The UML was established as a standard "to model systems (and not just software)," and thus applies to views about software architecture.

1 introduction yao-yuan chuang. 2 outline high performance computing scientific computing ...

Documents

computing systems

highperformance computing

computer systems

term super computing

teraflop slide

pflops slide

model systems

hpc systems