fall 2008 cs 668 parallel computing prof. fred annexstein [email protected] office hours: 11-1...
Post on 21-Dec-2015
216 views
TRANSCRIPT
Fall 2008CS 668
Parallel ComputingProf. Fred Annexstein
Office Hours: 11-1 MW or by appointment
Tel: 513-556-1807
Lecture 1: Welcome
• Goals of this course• Syllabus, policies, grading• Blackboard Resources• LINC Linux cluster• Introduction/Motivation for HPPC• Scope of the Problems in Parallel Computing
Goals
• Primary: – Provide an introduction to the computing systems,
programming approaches, common numerical and algorithmic methods used for high performance parallel computing
• Secondary:– Have an course meeting competency requirements of
RRSCS – Provide hands-on parallel programming experience
• Official Syllabus Available on Blackboard
• Textbook
Parallel Programming in C with MPI and OpenMP, Michael J. Quinn
Other Recommended Texts- Parallel Programming With Mpi, Peter Pacheco- Introduction to Parallel Computing: Design and Analysis
of Algorithms: Ananth Grama, Anshul Gupta, George Karpis, Vipin Kumar
- Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface by William Gropp
Workload/Grading
• Exams (1 or 2)– Graded 30% of Grade
• Written exercises (3-4)– May/may not be graded
• Programming Assignments (3-4)– May be done in groups of at most 2– MPI programming, performance measurement
• Research papers (1)– Discussion research questions, strengths, weaknesses,
interesting points, contemporary bibliography
• Final project (1)– Individual or Group programming project and report
Policies
• Missed Exams: – Missed exams can not be made up unless pre-
approved. Please see the instructor as soon as possible in the event of a conflict.
• Academic Honesty: – Plagiarism on assignments, quizzes or exams will not
be tolerated. See your student code of conduct (http://www.uc.edu/conduct/Code_of_Conduct.html) for more on the consequences of academic misconduct. There are no “small” offenses.
Blackboard
• Syllabus and my contact info
• Announcements
• Lecture slides
• Assignment handouts
• Web resources relevant to the course
• Discussion board
• Grades
What is the Ralph Regula School?
• The Ralph Regula School of Computational Science is a statewide, virtual school focused on computational science. It is a collaborative effort of the Ohio Board of Regents, Ohio Supercomputer Center, Ohio Learning Network and Ohio's colleges and universities. With funding from NSF, the school acts as a coordinating entity for a variety of computational science education activities aimed at making education in computational science available to students across Ohio, as well as to workers seeking continuing education about this technology.
• Website: http://www.rrscs.org
CS LINC Cluster
• Michal Kouril’s links– http://www.ececs.uc.edu/~kourilm/clusters/– See README file for instructions on running MPI
code on beowulf.linc.uc.edu• Accounts
– ECE/CS students should already have an account– I can request accounts for the non-ECE/CS students
• Access– Remote access only, the cluster is in the ECE/CS
server/machine room on the 8th floor of Rhodes, visible through windows in the 890’s hallway
Why HPPC?
• Who needs a roomful of computers anyway?
• My PC and XBOX run at GFLOP rates (Billion Floating Point Operations per second)
NCSA TeraGrid IA-64 Linux Cluster(http://www.ncsa.uiuc.edu/UserInfo/Resources/Hardware/TGIA64LinuxCluster/)
Needed by People who solve Science and Engineering problems• Materials / Superconductivity• Fluid Flow• Weather/Climate• Structural Deformation • Genetics / Protein interactions• Seismic
Many Research Projects in Natural Sciences and Engineering cannot exist without HPPC
Applications
• Videos – Applications in Physics and Geology
• Simulation of Large-Scale Structure of Universe http://www.youtube.com/watch?v=8C_dnP2fvxk
• Stability Simulation – http://www.youtube.com/watch?v=ZCMiLJOXrpc
• Super Volcano Movie - Show first 1:00 minute http://www.youtube.com/watch?v=unGODG7N1Bs
Why are the problems so large?
• 3-Dimensional– If you want to increase the level of resolution by factor
of 10, problem size increases by 103
• Many Length Scales (both time and space)– If you want to observe the interactions between very
small local phenomenon and larger more global phenomenon
• The number of relationships between data items grows quadraticly. – Example: human genome 3.2 G base pairs means
about 5,000,000,000,000,000,000=5E relations
How can you solve these problems?
• Take advantage of parallelism– Large problems generally have many operations
which can be performed concurrently
• Parallelism can be exploited at many levels by the computer hardware– Within the CPU core, multiple functional units,
pipelining– Within the Chip, many cores– On a node, multiple chips– In a system, many nodes
However….
• Parallelism has overheads– At the core and chip level the cost is
complexity and money– Most applications get only a fraction of peak
performance (10%-20%)– At the chip and node level, memory bus can
get saturated if too many cores– Between nodes, the communication
infrastructure is typically much slower than the CPU
Necessity Yields Modest Success
• Power of CPUs keeps growing Power of CPUs keeps growing exponentiallyexponentially
• Parallel programming environments Parallel programming environments changing very slowly – much harder than changing very slowly – much harder than sequentialsequential
Two standards have emergedTwo standards have emerged• MPI library, for processes that do not share MPI library, for processes that do not share
memorymemory• OpenMP directives, for processes that do OpenMP directives, for processes that do
share memoryshare memory
Why MPI?
• MPI = “Message Passing Interface”MPI = “Message Passing Interface”
• Standard specification for message-Standard specification for message-passing librariespassing libraries
• Very PortableVery Portable
• Libraries available on virtually all parallel Libraries available on virtually all parallel computerscomputers
• Free libraries also available for networks Free libraries also available for networks of workstations or commodity clustersof workstations or commodity clusters
Why OpenMP?
• OpenMP an application programming OpenMP an application programming interface (API) for shared-memory interface (API) for shared-memory systemssystems
• Based on model of creating and Based on model of creating and scheduling multi-threaded computations.scheduling multi-threaded computations.
• Supports higher performance parallel Supports higher performance parallel programming of symmetrical programming of symmetrical multiprocessorsmultiprocessors
What are the Costs?Commercial Parallel Systems• Relatively costly per processorRelatively costly per processor• Primitive programming environmentsPrimitive programming environments• Scientists looked for alternativeScientists looked for alternative
Beowulf Concept circa 1994• NASA project (written by Sterling and Becker)NASA project (written by Sterling and Becker)• Commodity processorsCommodity processors• Commodity interconnectCommodity interconnect• Linux operating systemLinux operating system• Message Passing Interface (MPI) libraryMessage Passing Interface (MPI) library• High performance/$ for certain applicationsHigh performance/$ for certain applications
How are they Programmed? Task Dependence Graph• Begin with Directed graphBegin with Directed graph• Vertices = tasks Edges = dependencesVertices = tasks Edges = dependences• Edges are removed as tasks completeEdges are removed as tasks complete
Data Parallelism• Independent tasks apply same operation to different elements of a Independent tasks apply same operation to different elements of a
data setdata set
Functional Parallelism• Independent tasks apply different operations to different data Independent tasks apply different operations to different data
elementselements
Pipelining• Divide a process into stagesDivide a process into stages• Produce and consume several items simultaneouslyProduce and consume several items simultaneously
Why not just use a Compiler?• Parallelizing compiler - Detect parallelism in sequential programParallelizing compiler - Detect parallelism in sequential program• Produce parallel executable programProduce parallel executable program
Advantages Advantages Can leverage millions of lines of existing serial programsCan leverage millions of lines of existing serial programs
• Saves time and labor- Requires no retraining of programmersSaves time and labor- Requires no retraining of programmers• Sequential programming easier than parallel programmingSequential programming easier than parallel programming
DisadvantagesDisadvantages• Parallelism may be irretrievably lost when programs written in Parallelism may be irretrievably lost when programs written in
sequential languagessequential languages• Simple example: Compute all partial sums in an arraySimple example: Compute all partial sums in an array• Performance of parallelizing compilers on broad range of Performance of parallelizing compilers on broad range of
applications still up in airapplications still up in air
Can we Extend Existing Languages?
Programmer can give directives or clues to the Programmer can give directives or clues to the complier about how to parallelizecomplier about how to parallelize
AdvantagesAdvantages• Easiest, quickest, and least expensiveEasiest, quickest, and least expensive• Allows existing compiler technology to be Allows existing compiler technology to be
leveragedleveraged• New libraries can be ready soon after new New libraries can be ready soon after new
parallel computers are availableparallel computers are availableDisadvantagesDisadvantages• Lack of compiler support to catch errorsLack of compiler support to catch errors• Easy to write programs that are difficult to debugEasy to write programs that are difficult to debug
Or Create New Parallel Languages?
AdvantagesAdvantages• Allows programmer to communicate parallelism Allows programmer to communicate parallelism
to compiler directlyto compiler directly• Improves probability that executable will achieve Improves probability that executable will achieve
high performancehigh performance
DisadvantagesDisadvantages• Requires development of new compilersRequires development of new compilers• New languages may not become standardsNew languages may not become standards• Programmer resistanceProgrammer resistance
Where are we in 2008?
• Performance makes Low-level approaches Performance makes Low-level approaches popularpopular
• Augment existing language with low-level Augment existing language with low-level parallel constructs and directivesparallel constructs and directives
• MPI and OpenMP are prime examplesMPI and OpenMP are prime examples
Advantages Advantages • EfficiencyEfficiency• PortabilityPortabilityDisadvantagesDisadvantages• More difficult to program and debugMore difficult to program and debug
Programming Assignment #1
• Log into beowulf.linc.uc.edu and run some simple sample programs.
Reading Assignment #1 on Blackboard