1 a look at library software for linear algebra: past, present, and future jack dongarra university...
Post on 11-Jan-2016
226 Views
Preview:
TRANSCRIPT
1
A Look at Library Software for A Look at Library Software for Linear Algebra: Past, Present, Linear Algebra: Past, Present, and Futureand Future
Jack Dongarra University of Tennessee
and Oak Ridge National Laboratory
2
1960’s1960’s 1961
– IBM Stretch delivered to LANL
1962– Virtual memory from U
Manchester, T. Kilburn 1964
– AEC urges manufacturers to look at ``radical new'' machine structures.
» This leads to CDC Star-100, TI ASC, and Illiac-IV.
– CDC 6600; S. Cray's design» Functional parallelism, leading
to RISC» (3 times faster than the IBM
Stretch)
1965– DEC ships first PDP-8 & IBM ships 360– First CS PhD U of Penn. Richard Wexelblat– Wilkinson’s The Algebraic Eigenvalue
Problem published 1966
– DARPA contract with U of I to build the ILLIAC IV
– Fortran 66 1967
– Forsythe & Moler published» Fortran, Algol and PLI
1969 – DARPANET work begins
» 4 computers connected UC-SB,UCLA, SRI and U of Utah
– CDC 7600» Pipelined architecture; 3.1 Mflop/s
– Unix developed Thompson and Ritchie
.
.
.
.
.
3
Wilkinson-Reinsch HandbookWilkinson-Reinsch Handbook “In general we have aimed to include only
algorithms that provide something approaching an optimal solution, this may be from the point of view of generality, elegance, speed, or economy of storage.”– Part 1 Linear System– Part 2 Algebraic Eigenvalue Problem
Before publishing Handbook– Virginia Klema and others at ANL began
translation of the Algol procedures into Fortran subroutines
4
1970’s1970’s 1970
– NATS Project conceived» Concept of certified MS and process
involved with production
– Purdue Math Software Symposium– NAG project begins
1971 – Handbook for Automatic Computation,
Vol II » Landmark in the development of
numerical algorithms and software» Basis for a number of software projects
EISPACK, a number of linear algebra routines in IMSL and the F chapters of NAG
– IBM 370/195» Pipelined architecture; Out of order
execution; 2.5 Mflop/s
– Intel 4004; 60 Kop/s– IMSL founded
1972– Cray Research founded– Intel 8008– 1/4 size Illiac IV installed NASA Ames
» 15 Mflop/s achieved; 64 processors
– Paper by S. Reddaway on massive bit-level parallelism
– EISPACK available» 150 installations; EISPACK Users' Guide» 5 versions, IBM, CDC, Univac, Honeywell
and PDP, distributed free via Argonne Code Center
– M. Flynn publishes paper on architectural taxonomy
– ARPANet» 37 computers connected
1973– BLAS report in SIGNUM
» Lawson, Hanson, &Krogh
.
.
.
.
.
5
NATS ProjectNATS Project National Activity for Testing Software
(NSF, Argonne, Texas and Stanford) Project to explore the problems of testing,
certifying, disseminating and maintaining quality math software.– First EISPACK, later FUNPACK
» Influenced other “PACK”s ELLPACK, FISKPACK, ITPACK,MINPACK,PDEPACK, QUADPACK,
SPARSPAK, ROSEPACK, TOOLPACK, TESTPACK,LINPACK, LAPACK, ScaLAPACK . . .
Key attributes of math software– reliability, robustness, structure, usability, and
validity
6
EISPACKEISPACK Algol versions of the algorithms written in Fortran
– Restructured to avoid underflow– Check user’s claims such as if matrix positive definite– Format programs in a unified fashion
» Burt Garbow
– Field test sites 1971 U of Michigan Summer Conference
– JHW algorithms & CBM software 1972 Software released via Argonne Code Center
– 5 versions Software certified in the sense that reports of poor or
incorrect performance “would gain the immediate attention from the developers”
7
EISPACKEISPACK EISPAC Control Program, Boyle
– One interface allowed access to whole package on IBM
Argonne’s interactive RESCUE system– Allowed us to easily manipulate 10s of 1000’s of
lines Generalizer/Selector
– Convert IBM version to general form– Selector extract appropriate version
1976 Extensions to package ready – EISPACK II
8
1970’s continued1970’s continued 1974
– Intel 8080– Level 1 BLAS activity started by
community; Purdue 5/74– LINPACK meeting in summer ANL
1975– First issue of TOMS– Second LINPACK meeting ANL
» lay the groundwork and hammer out what was and was not to be included in the package. Proposal submitted to the NSF
1976 – Cray 1 - model for vector computing
» 4 Mflop/s in 79; 12 Mflop/s in 83; 27 Mflop/s in 89
– LINPACK work during summer at ANL– EISPACK Second edition of User's Guide
1977 – DEC VAX 11/780; super mini
» .14 Mflop/s; 4.3 GB virtual memory
– LINPACK test software developed &sent– EISPACK second release– IEEE Arithmetic standard meetings
» paper by Palmer on INTEL std for fl pt
1978– Fortran 77– LINPACK software released
» Sent to NESC and IMSL for distribution
1979 – John Cocke designs 801– ICL DAP delivered to QMC, London– Level 1 BLAS published/released– LINPACK Users' Guide
» Appendix: 17 machines PDP-10 to Cray-1
.
.
.
.
9
Basic Linear Algebra SubprogramsBasic Linear Algebra Subprograms BLAS 1973-1977 Consensus on:
– Names– Calling sequences– Functional Descriptions– Low level linear algebra
operations Success results from
– Extensive Public Involvement
– Careful consideration of implications
A design tool software in numerical linear algebra
Improve readability and aid documentation
Aid modularity and maintenance, and improve robustness of software calling the BLAS
Improve portability, without sacrificing efficiency, through standardization
10
CRAY and VAXCRAY and VAX
VAXination of groups and departments
Cray’s introduction of vector computing
Both had a significant impact on scientific computing
11
LINPACKLINPACK June 1974, ANL
– Jim Pool’s meeting February 1975, ANL
– groundwork for project January 1976,
– funded by NSF and DOE: ANL/UNM/UM/UCSD Summer 1976 - Summer 1977 Fall 1977
– Software to 26 testsites December 1978
– Software released, NESC and IMSL
LINPACKLINPACK Research into
mechanics of software production. Provide a yearstick against which future
software would be measured. Produce library used by people and those
that wished to modify/extend software to handle special problems.
Hope would be used in classroom Machine independent and efficient
– No mix mode arithmetic
L I N P A C K
I N P A C K
N P A C K
P A C K
A C K
C K
K
F
H
GGGGGGGG
I
K
JJJJJJJJ
1 1
1 1
1 1
1 1
1 1
1 1
1
/ /
/ /
/ /
/ /
/ /
/ /
/
L L
I I
N N
P P
A A
C C
K
F
H
GGGGGGGG
I
K
JJJJJJJJ
13
LINPACK Efficiency LINPACK Efficiency Condition estimator Inner loops via BLAS Column access Unrolling loops (done for the BLAS) BABE Algorithm for tridiagonal matrices TAMPR system allowed easy generation
of versions LINPACK Benchmark
– Today reports machines from Cray T90 to Palm Pilot» (1.0 Gflop/s to 1.4 Kflop/s)
for i = 1:n:4,
end
y y x
y y x
y y x
y y x
i i i
i i i
i i i
i i i
;
;
;
;
1 1 1
2 2 2
3 3 3
0
0
14
1980’s Vector Computing 1980’s Vector Computing (Parallel Processing)(Parallel Processing)
1980– Total computers in use in the US
exceeds 1M– CDC Introduces Cyber 205
1981– IBM introduces the PC
» Intel 8088/DOS
– BBN Butterfly delivered– FPS delivers FPS-164
» Start of mini-supercomputer market
– SGI founded by Jim Clark and others– Loop unrolling at outer level for
data locality and parallelism» Amounts to matrix-vector operations
– Cuppen's method for Symmetric Eigenvalue D&C published
» Talk at Oxford Gatlinburg (1980)
1982– Illiac IV decommissioned– Steve Chen's group at Cray produces X-MP– First Denelcor HEP installed (.21 Mflop/s)– Sun Microsys, Convex and Alliant founded
1983– Total computers in use in the US exceed 10 M– DARPA starts Strategic Computing Initiative
» Helps fund Thinking Machines, BBN, WARP
– Cosmic Cube hypercube running at Caltech» John Palmer, after seeing Caltech machine,
leaves Intel to found Ncube
– Encore , Sequent, TMC, SCS, Myrias founded– Cray 2 introduced– NEC SX-1 and SX-2, Fujitsu ships VP-200– ETA System spun off from CDC– Golub & Van Loan published
.
.
.
.
.
.
.
.
.
15
1980’s continued1980’s continued 1984
– NSFNET; 5000 computers;56 Kb/s lines
– MathWorks founded– EISPACK third release– Netlib begins 1/3/84– Level 2 BLAS activity started
» Gatlinburg(Waterloo), Purdue, SIAM
– Intel Scientific Computers started by J. Rattner
» Produce commerical hypercube
– Cray X-MP 1 processor, 21 Mflop/s Linpack
– Multiflow founded by J. Fisher; VLIW architecture
– Apple introduces Mac &IBM introduces PC AT
– IJK paper
1985– IEEE Standard 754 for floating point– IBM delivers 3090 vector; 16 Mflop/s
Linpack, 138 Peak– TMC demos CM1 to DARPA– Intel produces first iPSC/1
Hypercube» 80286 connected via Ethernet
controllers
– Fujitsu VP-400; NEC SX-2; Cray 2; Convex C1
– Ncube/10 ; .1 Mflop/s Linpack 1 processor
– FPS-264; 5.9 Mflop/s Linpack 38 Peak– IBM begins RP2 project– Stellar, Poduska & Ardent, Michels
Supertek Computer founded– Denelcor closes doors
.
.
.
.
.
16
1980’s 1980’s (Lost Decade for Parallel Software)(Lost Decade for Parallel Software) 1986
– # of computers in US exceeds 30 M– TMC ships CM-1; 64K 1 bit
processors– Cray X-MP– IBM and MIPS release first RISC WS
1987– ETA Systems family of
supercomputers– Sun Microsystems introduces its
first RISC WS– IBM invests in Steve Chen's SSI– Cray Y-MP– First NA-DIGEST– Level 3 BLAS work begun– LAPACK: Prospectus Development of
a LA Library for HPC
1988– AMT delivers first re-engineered DAP– Intel produces iPSC/2– Stellar and Ardent begin delivering
single user graphics workstations– Level 2 BLAS paper published
1989– # of computers in the US > 50M– Stellar and Ardent merge, forming
Stardent– S. Cray leaves Cray Research to form
Cray Computer– Ncube 2nd generation machine– ETA out of business– Intel 80486 and i860 ; 1 M
transistors» i860 RISC & 64 bit floating point
.
.
.
.
.
.
17
EISPACK 3 and BLAS 2 & 3EISPACK 3 and BLAS 2 & 3 Machine independence for
EISPACK Reduce the possibility of
overflow and underflow. Mods to the Algol from S.
Hammarling Rewrite reductions to
tridiagonal form to involve sequential access to memory
“Official” Double Precision version
Inverse iteration routines modified to reduce the size for reorthogonalization.
BLAS (Level 1) vector operations provide for too much data movement.
Community effort to define extensions– Matrix-vector ops– Matrix-matrix ops
18
Netlib - Mathematical Netlib - Mathematical Software and DataSoftware and Data Began in 1985
– JD and Eric Grosse, AT&T Bell Labs Motivated by the need for cost-effective, timely distribution
of high-quality mathematical software to the community. Designed to send, by return electronic mail, requested items. Automatic mechanism for electronic dissemination of freely
available software.– Still in use and growing– Mirrored at 9 sites around the world
Moderated collection /Distributed maintenance
NA-DIGEST and NA-Net– Gene Golub, Mark Kent and Cleve Moler
19
Netlib Growth Netlib Growth
0
2000000
4000000
6000000
8000000
10000000
Nu
mb
er o
f re
qu
ests
1985
1987
1989
1991
1993
1995
1997
Year
gopher
http
ftp
xnetlib
Just over 6,000 in 1985 Over 29,000,000 totalOver 9 Million hits in 19975.4 Million so far in 1998
LAPACK Best Seller 1.6 M hits
20
1990’s1990’s 1990
– Internet– World Wide Web
– Motorola introduces 68040– NEC ships SX-3; First parallel
Japanese parallel vector supercomputer
– IBM announces RS/6000 family» has FMA instruction
– Intel hypercube based on 860 chip» 128 processors
– Alliant delivers FX/2800 based on i860
– Fujitsu VP-2600– PVM project started– Level 3 BLAS Published
1991– Stardent to sell business and close– Cray C-90– Kendall Square Research delivers 32
processor KSR-1– TMC produces CM-200 and
announces CM-5 MIMD computer– DEC announces the Alpha– TMC produces the first CM-5– Fortran 90– Workshop to consider Message
Passing Standard, beginnings of MPI» Community effort
– Xnetlib running 1992
– LAPACK software released & Users' Guide Published
.
.
.
.
21
Architectures
328312
69
118
230
227
156
265
362
140
344
246
71
134
81130
2766
10190
0
50
100
150
200
250
300
350
400
# S
yste
ms
MPP PVP SMP
22
Parallel Processing Comes of AgeParallel Processing Comes of Age "There are three rules for
programming parallel computers.
We just don't know what they are yet." -- Gary Montry
“Embarrassingly Parallel”, Cleve Moler
“Humiliatingly Parallel”
23
Memory Hierarchy Memory Hierarchy and LAPACKand LAPACK ijk - implementations
Effects order in which data referenced; some better at allowing data to keep in higher levels of memory hierarchy.
Applies for matrix multiply, reductions to condensed form– May do slightly more
flops– Up to 3 times faster
for _ = 1:n;
for _ = 1:n;
for _ = 1:n;
end
end
end
a a b ci j i j i k k j, , , ,
24
BLAS MemoryRefs
Flops Flops/MemoryRefs
Level 1y=y+x
3n 2n 2/3
Level 2y=y+Ax
n2 2n2 2
Level 3C=C+AB
4n2 2n3 n/2
Why Higher Level BLAS?Why Higher Level BLAS? Can only do
arithmetic on data at the top of the hierarchy
Higher level BLAS lets us do this Development of
blocked algorithms important for performance
IBM RS/6000-590 (66 MHz, 264 Mflop/s Peak)
0
50
100
150
200
250
10 100 200 300 400 500Order of vector/Matrices
Mfl
op
/s
Level 3 BLAS
Level 2 BLAS
Level 1 BLAS
25
History of Block Partitioned History of Block Partitioned AlgorithmsAlgorithms
Ideas not new. Early algorithms involved use
of small main memory using tapes as secondary storage.
Recent work centers on use of vector registers, level 1 and 2 cache, main memory, and “out of core” memory.
26
LAPACKLAPACK Linear Algebra library in Fortran 77
– Solution of systems of equations– Solution of eigenvalue problems
Combine algorithms from LINPACK and EISPACK into a single package
Block algorithms– Efficient on a wide range of computers
» RISC, Vector, SMPs
User interface similar to LINPACK– Single, Double, Complex, Double Complex
Built on the Level 1, 2, and 3 BLAS HP-48G to CRAY T-90
Cray Y-MPCholesky, n=500
1 Proc 8 procs
j-varientLINPACK
72 72
j-varientLevel 2 BLAS
251 378
j-varientLevel 3 BLAS
287 1225
k-varientLevel 3 BLAS
290 1414
Mflop/s
27Rate
ops
Time
23
3 2n n
28
1990’s continued1990’s continued 1993
– Intel Pentium system start to ship– ScaLAPACK Prototype software
released » First portable library for distributed
memory machines» Intel, TMC and workstations using PVM
– PVM 3.0 available 1994
– MPI-1 Finished 1995
– Templates project 1996
– Internet; 34M Users– Nintendo 64
» More computing power than a Cray 1 and much much better graphics
1997– MPI-2 Finished – Fortran 95
1998– Issues of parallel and
numerical stability– Divide time– DSM architectures– "New" Algorithms
» Chaotic iteration» Sparse LU w/o pivoting» Pipeline HQR» Graph partitioning» Algorithmic bombardment
.
.
.
.
29
Templates ProjectTemplates Project Iterative methods for large sparse
systems– Communicate to HPC community “State of
the Art” algorithms– Subtle algorithm issues addressed, i.e.
convergence, preconditions, data structures
– Performance and parallelism considerations
– Gave the computational scientists algorithms in form they wanted.
30
ScaLAPACKScaLAPACK Library of software for dense & banded
»Sparse direct being developed
Distributed Memory - Message Passing– PVM and MPI
MIMD Computers, Networks of Workstations, and Clumps of SMPs
SPMD Fortran 77 with object based design Built on various modules
– PBLAS (BLACS and BLAS)» PVM, MPI, IBM SP, CRI T3, Intel, TMC» Provides right level of notation.
31
High-Performance Computing High-Performance Computing Directions Directions
Move toward shared memory– SMPs and Distributed Shared Memory– Shared address space w/deep memory hierarchy
Clustering of shared memory machines for scalability– Emergence of PC commodity systems
» Pentium based, NT or Linux driven At UTK cluster of 14 (dual) Pentium based 7.2 Gflop/s
Efficiency of message passing and data parallel programming– Helped by standards efforts such as PVM, MPI and HPF
Complementing “Supercomputing” with Metacomputing Computational Grid
32
Heterogeneous ComputingHeterogeneous Computing Heterogeneity introduces new bugs in parallel code Slightly different fl pt can make data dependent
branches go different ways when we expect identical behavior.
A “correct” algorithm on a network of identical workstations may fail if a slightly different machine is introduced.
Some easy to fix (compare s < tol on 1 proc and broadcast results)
Some hard to fix (handling denorms; getting the same answer independent of # of procs)
33
Java - Java - For Numerical Computations?For Numerical Computations?
Java likely to be a dominant language. Provides for machine independent code. C++ like language No pointers, goto’s, overloading arith
ops, or memory deallocation Portability achieved via abstract
machine Java is a convenient user interface
builder which allows one to quickly develop customized interfaces.
34
Network Enabled Network Enabled ServersServers
Allow networked resources to be integrated into the desktop.
Many hosts, co-existing in a loose confederation tied together with high-speed links.
Users have the illusion of a very powerful computer on the desk.
Locate and “deliver” software or solutions to the user in a directly usable and “conventional” form.
Part of the motivation software maintenance
35
Future: Petaflops ( fl pt ops/s)Future: Petaflops ( fl pt ops/s)
A Pflop for 1 second a typical workstation computing for 1 year.
From an algorithmic standpoint– concurrency– data locality– latency & sync– floating point accuracy
May be feasible and “affordable” by the year 2010
1015
– dynamic redistribution of workload
– new language and constructs
– role of numerical libraries– algorithm adaptation to
hardware failure
Today flops for our workstations 1015
36
SummarySummary
As a community we have a lot to be proud of in terms of the algorithms and software we have produced.– generality, elegance, speed, or
economy of storage Software still being used in many
cases 30 years after
37
top related