tsf and abstract interfaces in trilinos tsfcore, tsfextended … roscoe a. bartlett department 9211:...
TRANSCRIPT
TSF and Abstract Interfaces in Trilinos
TSFCore, TSFExtended …
Roscoe A. Bartlett
Department 9211: Optimization and Uncertainty Estimation
Sandia National Laboratories
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Categories of Abstract Problems and Abstract Algorithms
Linear Problems:
Linear equations:
Eigen problems:
Nonlinear Problems:
Nonlinear equations:
Stability analysis:
Transient Nonlinear Problems:
DAEs/ODEs:
Optimization Problems:
Unconstrained:
Constrained:
Trilinos Packages
Belos
Anasazi
NOX
LOCA
Rhythms, TOX ???
MOOCHO
Common Environments for Scientific Computing
Serial / SMP (symmetric multi-processor)
• All data stored in RAM in a single local process Code
Data
Processor
Out of Core
• Data stored in file(s) (too big to fit in RAM)Processor Disk
Code Data SPMD (Single program multiple data)
• Same code on each processor but different data
Code
Data(0)
Proc 0
Code
Data(1)
Code
Data(N-1)
MPP (massively parallel processors)
…
Proc 1 Proc N-1
Introducing Abstract Numerical Algorithms
An ANA is a numerical algorithm that can be expressed abstractly solely in terms of vectors, vector spaces, and linear operators (i.e. not with direct element access)
Example Linear ANA (LANA) : Linear Conjugate Gradients
scalar product<x,y> defined by vector space
vector-vector operations
linear operator applications
Scalar operations
Types of operations Types of objects
What is an abstract numerical algorithm (ANA)?
Linear Conjugate Gradient Algorithm
Examples of Nonlinear Abstract Numerical Algorithms
Class of Numerical Problem Example ANA
Nonlinear equations Newton’s method (e.g. NOX, MOOCHO)
Initial Value DAE/ODE Backward Euler method (e.g. Rhythms, TOX???)
Linear equations GMRES (e.g. Belos)
Requires
Requires
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Unified Modeling Language (UML)
Good tutorial on object-orientation development process and short (about 200 pages) introduction to the UML.
Unified Modeling Language
Graphical language for designing, documenting and communicating designs object-oriented software
Standardized by the Object Management Group (OMG) and backed by a large group of software companies
Allows the specification and description of object-oriented software at various levels of detail
Supported by many different CASE tools for
Designing OO software
Documenting OO software
Reverse engineering OO software
Creating skeleton code for OO software
Refactoring OO software
eFunc(in c: C)
class BaseA {public: virtual ~BaseA() {} virtual int aFunc() const = 0; virtual C& c() = 0; …};
class DerivedB : public BaseA {public: …private: std::vector<BaseD*> d_;};
class C { … };
class BaseD { … };
class DerivedE : public BaseE {public: void eFunc( const C& c ); …};
DerivedB
aFunc(): int
Overview of Basic UML Notation for Object Orientation (OO)
BaseA Cc
DerivedE
d
1…*
BaseD
Simple UML Class Diagram Compatible C++ declarations
Inheritance
Abstract class(italics)
Concrete class(non-italics)
Abstract member function(italics)
Association
Member function implementation
(non-italics)
Association
Multiplicity
Classes
Relationshipss
Dependency
Abstract classand inheritance
(italics)
Design Patterns : “Standard” Vocabulary for OO
Some of the Design Patterns used in TSF
Creational Patterns
“Abstract Factory”
“Prototype”
Structural Patterns
“Adapter”
“Composite”
“Decorator”
“Proxy”
Behavior Patterns
“Strategy”
“Visitor”
Design Patterns in UML Notation:
http://exciton.cs.oberlin.edu/javaresources/DesignPatterns/default.htm
Good tutorial on object-orientation and a catalog of 23 fundamental design patterns of object-oriented software
“Common problems have common solutions!”
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Software Componentization and TSF Interfaces
2) LAL : Linear Algebra Library (e.g. vectors, sparse matrices, sparse factorizations, preconditioners)
ANA
APP
ANA/APP Iterface
ANA Vector Interface
ANA Linear Operator Interface
1) ANA : Abstract Numerical Algorithm (e.g. linear solvers, eigen solvers, nonlinear solvers, stability analysis, uncertainty quantification, transient solvers, optimization etc.)
3) APP : Application (the model: physics, discretization method etc.)
Example Trilinos Packages:• Belos (linear solvers)• Anasazi (eigen solvers)• NOX (nonlinear equations)• Rhythms, TOX??? (ODEs,DAEs)• MOOCHO (Optimization)• …
Example Trilinos Packages:• Epetra/Tpetra (Mat,Vec)• Ifpack, AztecOO, ML (Preconditioners)• Meros (Preconditioners)• Pliris (Interface to direct solvers)• Amesos (Direct solvers)• Komplex (Complex/Real forms)• …Types of Software Components
TSFCoreANA Interfaces to Linear Algebra
TSFExtendedAPP to LAL Interfaces TSFExtended
LAL to LAL Interfaces
TSFCore::Nonlin ???
Examples:• SIERRA• NEVADA• Xyce• Sundance• …
LAL
Matrix Preconditioner
Vector
ANA software allows for a much more abstract interface than APP or LAL software!
Different Platform Configurations for Running ANAs
SPMDANA
LAL
Proc 0
APP
MPP
ANA
LAL
Proc 1
APP ANA
LAL
Proc N-1
APP
…
Master/SlaveANA
LAL
Proc 0 (master)
APP
MPP
LAL
Proc 1
APP
…
Client/ServerMaster/Slave
LAL
Proc N-1
APP
MPP Server
LAL
Proc 1
APP…
LAL
Proc N-1
APP
LAL
Proc 0
APP
Client
ANA
If a code can not run in these modes it is not an ANA code!
Requirements for Abstract Numerical Algorithms and TSF
An important consideration Scientific computing is computationally expensive!
Abstract Interfaces for Abstract Numerical Algorithms using TSF must:
• Be portable to all ASC (advanced scientific computing) computer platforms
• Provide for stable and accurate numerics
• Result in algorithms with near-optimal storage and runtime performance– Scientific computing is expensive!
An important ideal A customized hand-code algorithm in Fortran 77 should not provide significantsignificant improvements in storage requirements, speed or numerical stability!
Abstract Interfaces for Abstract Numerical Algorithms using TSF should:
• Be minimal but complete (good object-oriented design principle)
• Support a variety of computing platforms and configurations– i.e. Serial/SMP, Out-of-Core, SPMD, master/slave and client/server
• Be (relatively) easy to provide new implementations (Subjective!!!)
• Not be too complicated (Subjective!!!)
An important ideal Object-oriented “overhead” should be constant and not increase as the problem size increases.
LinearOpVectorSpace
OpBase
Vector
MultiVector
1
columns1..*
RTOpT
rangedomain
space
Fundamental TSFCore Linear Algebra Interfaces
An operator knows its domain and range VectorSpaces
A MultiVector has a collection of column vectors
A MultiVector is a linear operator
A Vector is a MultiVector and therefore is a linear operator
A Vector knows its VectorSpace
A linear operator can be applied to Vectors and MultiVectors
A linear operator is a kind of operator
Trilinos TSFCore website
Fundamental TSFCore Linear Algebra Interfaces
createMember() : VectorcreateMembers(in numMembers : int) : MultiVectorisCompatible(in vecSpc : VectorSpace) : boolscalarProd(in x : Vector, in y : Vector) : Scalar
dim : int
TSFCore::VectorSpace
applyOp(in op : RTOpT, inout ...)
TSFCore::Vector
apply_op(inout ...)reduce_reduct_objs(inout ...)
RTOpPack::RTOpT
applyOp(in op : RTOpT, inout ...)subView(in col_rng : Range1D) : MultiVectorsubView(in numCols : int, in cols[1..numCols] : int) : MultiVector
TSFCore::MultiVector
opSupported(in M_trans) : bool
TSFCore::OpBase
1
columns1..*
«create»
«create»
domain
space
apply(in M_trans, in x : Vector, inout y : Vector, in ...)apply(in M_trans, in X : MultiVector, inout Y : MultiVector, in ...)
TSFCore::LinearOp
createVecSpc(in dim : int) : VectorSpace
TSFCore::VectorSpaceFactory
«create»
smallVecSpcFcty
range
VectorSpaces create Vectors and MultiVectors!
Vector and MultiVector versions of apply(…) must agree with scalar product:
Adjoints supported but are optional!
Only one vector operation!
Scalar product <x,y> defined by vector space
Creates domain spaces for MultiVector objects
Interface that supports all element-wise vector reduction and transformation operations!
VectorSpace defines compatibility of vectors and multi-vectors
Creation of arbitrary subviews of columns
DRu,vAu,Av H
What’s the Big Deal about Vector-Vector Operations?
Examples from OOQP (Gertz, Wright)
y y x i ni i i / , ...1y y x z i ni i i i , ...1
yy y y yy y y y
y y yi ni
i i
i i
i
min min
max max
min max
, ...ififif0
1 dx:max
Example from TRICE (Dennis, Heinkenschloss, Vicente)
d
b u w bw b
u a w aw a
i ni
i i i
i i
i i i
i i
( ) and and
( ) and . and .
, ...
/
/
1 2
1 2
01 0
01 0
1
ifififif
Example from IPOPT (Waechter)
Ui
Li
Ui
Lii
U
Li
Li
Ui
Lii
L
iU
iiU
iL
iiL
iU
iL
Li
UiL
i
i
xxxxx
xxxxxwhere
ni
xxx
xxx
xxxx
x
x
,maxˆ
,minˆ:
...1,
ˆifˆ
ˆifˆ
ˆˆif2 Currently in MOOCHO :
> 40 vector operations!
Many different and unusual vector operations are needed by interior point methods for optimization!
Vector Reduction/Transformation Operators Defined
Reduction/Transformation Operators (RTOp) Defined
z 1i … z q
i opt( i , v 1i … v p
i , z 1i … z q
i ) element-wise transformation opr( i , v 1
i … v pi , z 1
i … z qi ) element-wise reduction
2 oprr( 1 , 2 ) reduction of intermediate reduction objects
• v 1 … v p R n : p non-mutable (constant) input vectors
• z 1 … z q
R n : q mutable (non-constant) input/output vectors• : reduction target object (many be non-scalar (e.g. {yk ,k}), or NULL)
Key to Optimal Performance
• opt(…) and opr(…) applied to entire sets of subvectors (i = a…b) independently:z 1
a:b … z qa:b , op( a, b , v 1
a:b … v pa:b , z 1
a:b … z qa:b , )
• Communication between sets of subvectors only for NULL, oprr( 1 , 2 ) 2
RTOpT
apply_op(in sv[1..p], inout sz[1…q], inout beta )
ROpDotProd
apply_op(… )
TOpAssignVectors
apply_op(… )
…
Vector
applyOp( in : RTOpT, apply_op(in v[1..p] , inout z[1…q], inout beta)
SerialVectorStd
applyOp( … )
MPIVectorStd
applyOp( … )
Software Implementation (see RTOp paper on Trilinos website “Publications”)
• “Visitor” design pattern (a.k.a. function forwarding)
…
Technicalities about TSFCore Software
All interfaces are templated on Scalar type Currently supported types: real (float, double), complex (std::complex<float>,
std::complex<double>), extended precision (GNU MP mpf_class) Teuchos::ScalarTraits<> used for all math functions Teuchos::RawMPITraits<> and Teuchos::PrimativeTypeTraits<> used for MPI
Smart reference-counted pointer class Teuchos::RefCountPtr<> used for all dynamic memory management
C++ exception handling mechanism used to throw exceptions derived from std::exception TSFCore uses const everywhere religiously often resulting in const and non-const versions of
many operations Many operations have default implementations based on very few pure virtual methods RTOpT operators (and wrapper functions) are provided for many common level-1 vector and multi-
vector operations Supported platforms (Includes all SIERRA and ICC platforms): Yey Teuchos!
Linux/g++: g++ 3.1 … 3.3.x, software.sandia.gov, CS LAN DEC: Compaq C++ V6.5-014, stratus.sandia.gov, CS LAN Linux/Intel: Intel C++ version 7.1, liberty.sandia.gov, ICC Sun: Forte Developer 7 C++ 5.4 Patch 111715-14, sass3276a.sandia.gov, EngSci LAN SGI: MIPSpro Compilers: Version 7.4.2m, sasg1099.sandia.gov, EngSci LAN IBM: VisualAge C++ Professional for AIX, Version 7, saia028.sandia.gov, EngSci LAN (not yet) Janus (ASC Red): ciCC = pgCC Rel 3.1-4i, sasn100.sandia.gov, EngSci LAN
TSFCore Interface C++ Specification (pure virtual functions)
template<class Scalar> class VectorSpace {public: virtual Index dim() const = 0; virtual bool isCompatible( const VectorSpace<Scalar>& vecSpc ) const = 0; virtual Teuchos::RefCountPtr< Vector<Scalar> > createMember() const = 0; …};
template<class Scalar> class OpBase {public: virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > domain() const = 0; virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > range() const = 0; virtual bool opSupported(ETransp M_trans) const;};
template<class Scalar> class LinearOp : virtual public OpBase<Scalar> {public: virtual void apply( const ETransp M_trans, const Vector<Scalar> &x ,Vector<Scalar> *y ,const Scalar alpha=1.0 ,const Scalar beta=0.0 ) const = 0; …};
template<class Scalar> class MultiVector : virtual public LinearOp<Scalar> {public: virtual Teuchos::RefCountPtr<Vector<Scalar> > col(Index j) = 0; …};
template<class Scalar> class Vector : virtual public MultiVector<Scalar> {public: virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > space() const = 0; virtual void applyOp( const RTOpPack::RTOpT<Scalar> &op, const size_t num_vecs ,const Vector<Scalar>* vecs[], const size_t num_targ_vecs ,Vector<Scalar>* targ_vecs[] ,RTOpPack::ReductTarget *reduct_obj ,const Index first_ele ,const Index sub_dim ,const Index global_offset ) const = 0; …};
Shows only pure virtual functions on interfaces that must be implemented
Default MultiVector implementation based on a Vector implementation
Default Serial Vector, VectorSpace and VectorSpaceFactory implementation for domain space of MultiVectors for default MultiVectors
Only two nontrivial functions to override if MultiVectors not needed!
– LinearOp::apply(…)
– Vector::applyOp(…)
Providing an optimized MultiVector implementation requires overriding more functions
However, most concrete subclasses should not derive from these base cases directly!
TSFCore Interface C++ Specification (explicit element access)
template<class Scalar>class MultiVector : virtual public LinearOp<Scalar> {public: … virtual void getSubMultiVector( const Range1D &rowRng, const Range1D &colRng ,RTOpPack::SubMultiVectorT<Scalar> *sub_mv ) const; virtual void freeSubMultiVector( RTOpPack::SubMultiVectorT<Scalar>* sub_mv ) const; virtual void getSubMultiVector( const Range1D &rowRng, const Range1D &colRng ,RTOpPack::MutableSubMultiVectorT<Scalar> *sub_mv ); virtual void commitSubMultiVector( RTOpPack::MutableSubMultiVectorT<Scalar>* sub_mv ); …};
template<class Scalar>class Vector : virtual public MultiVector<Scalar> {public: … virtual void getSubVector( const Range1D& rng, RTOpPack::SubVectorT<Scalar>* sub_vec ) const; virtual void freeSubVector( RTOpPack::SubVectorT<Scalar>* sub_vec ) const; virtual void getSubVector( const Range1D& rng, RTOpPack::MutableSubVectorT<Scalar>* sub_vec ); virtual void commitSubVector( RTOpPack::MutableSubVectorT<Scalar>* sub_vec ); virtual void setSubVector( const RTOpPack::SparseSubVectorT<Scalar>& sub_vec ); …};
• Explicit sub-views are requested and released in separate function calls (allows creation of temporary storage data with copies)
• All have default implementations in terms of advanced RTOp subclasses
• In general, explicit access to Vector and MultiVector elements is an extremely bad thing to do (i.e. client/server), however:
• Use cases where explicit element access is needed or is convenient– Debugging small problems (i.e. printing Vector or MultiVector elements)
– Vectors and MultiVectors created from the domain space of a MultiVector (i.e. block GMRES)
– Automatic interoperability of serial and MPI SPMD Vector and MultiVector objects.
Three Use Cases for Fundamental TSFCore Interfaces
1. TSFCore as interoperability layer for linear ANA objects (i.e. linear operators, vectors, multi-vectors, reduction/transformation operators)
2. Development of concrete TSFCore implementation subclasses
1. Support for decoupling application-defined scalar products form concrete vector spaces
2. Support for concrete implementations
1. Support and concrete subclasses for serial shared-memory platforms
2. Support and concrete subclasses for MPI SPMD distributed memory platforms
3. Concrete “adatper” subclasses for Epetra objects
3. Using TSFCore objects for the development of ANAs
1. C++ wrapper functions for prewritten RTOpT subclasses
e.g. norm(x), assign(&y,x) …
2. Composite objects:
1. Product vectors, multi-vectors and vector spaces
e.g. x = [ v1; v2; v3; … ; vn ];
2. Composite linear operators
1. Additive linear operators: e.g. M = A + B + C + D;
2. Multiplicative linear operators: e.g. M = A * B * C * D;
3. Block linear linear operators: e.g. M = [ A, B ; C, D ];
3. Handle classes for operator overloading (currently in TSFExtended)
e.g. y = alpha*trans(A)*x + gamma*B*z + beta*y
Trilinos TSFCore website
TSFCore Subclasses for Serial and MPI Platforms
Support base classes for serial vector space, vector and multi-vector subclasses SerialVectorSpaceBase, SerialVectorBase, SerialMultiVectorBase Provide complete implementation of all functions (utilizes level-3 BLAS from Teuchos) in terms of
explicit element access provided by subclasses Universal interoperability of Serial Vectors and MultiVectors:
All Vector and MultiVector objects that return VectorSpace objects with isInCore()==true and dim()==dim() are automatically compatible through explicit element sub-views!
Standard concrete implementations of serial vector space, vector and multi-vector subclasses SerialVectorSpaceStd, SerialVectorStd, SerialMultiVectorStd Provide general implementations that are very efficient default implementations Note: Most serial applications only need to implement linear operators (e.g. derived from
SerialLinearOpBase) Support base classes for MPI SPMD vector space, vector and multi-vector subclasses
MPIVectorSpaceBase, MPIVectorBase, MPIMultiVectorBase Provide complete implementation of all functions (utilizes level-3 BLAS from Teuchos) in terms of
explicit element access provided by subclasses Universal interoperability of MPI SPMD Vectors and MultiVectors:
All Vector and MultiVector objects that return VectorSpace objects that support the MPIVectorSpaceBase interface with same numbers of local elements are automatically compatible through explicit element sub-views!
Standard concrete implementations of MPI SPMD vector space, vector and multi-vector subclasses MPIVectorSpaceStd, MPIVectorStd, MPIMultiVectorStd Provide general implementations that are very efficient default implementations Note: Most MPI applications only need to implement linear operators (e.g. derived from
MPILinearOpBase)
TSF Composite ANA Linear Algebra Subclasses
VectorSpaceblocks
1…m
ProductVectorSpace
Vectorblocks
1…m
ProductVector
<<create>>
“Composite” subclasses allow a collection of objects to be manipulated as one object Product vector spaces and product vectors:
Product vector spaces: Product vectors:
Block linear operator:
Multiplicative (composed) linear operator:
Additive (summation) linear operator:
blockRange
blockDomain LinearOp
BlockLinearOp
blocks
1…*
MultiplicativeLinearOp
1…*
AdditiveLinearOp
1…*
Extending Functionality Beyond TSFCore
Question: How to extend functionality beyond TSFCore interfaces?
Examples of extended functionality Explicit matrix transposition: i.e. A = AT
Preconditioner creation: i.e. A P, where P A-1
LinearOp-LinearOp multiplication: i.e. C = A * B, where C is formed explicitly
Possible approaches to extend functionality in an ANA
Extended mix-in interfaces inherited by concrete linear algebra subclasses Explicit matrix transposition : e.g. TSFExtended::ExplicitlyTransposableOp Preconditioner creation: e.g. TSFExtended::ILUFactorizableOp LinearOp-LinearOp multiplication: e.g. through TSFExtended::RowAccessableOp (Not an ANA!)
“Strategy” interfaces and subclasses that are external from linear algebra subclasses Example: Some ANA needs LinearOp-LinearOp multiplication C = A * B with A and B being
Epetra_VbrMatrix objects with large element blocks (level-3 BLAS) and with output C being a Epetra_CrsMatrix (EpetraExt?)!
TSFCore::LinearOp
TSFCore::EpetraLinearOp
Epetra_Operator
Epetra_CrsMatrix Epetra_VbrMatrix
LinearOpMultiplierStrategy
EpetraVbrToCrsMultiplier
SomeANA
multiply( in A, in B, out C)
Strategy interfaces offer the greater flexibility, better efficiency and preserve ANAs!
Dependencies between TSF-related Software
“ANA to linear algebra” handles for operator
overloading(e.g. y = A*x)(currently in
TSFExtended)
“ANA to linear algebra” composite subclasses
(e.g. ProductVector)(currently in TSFCore and
TSFExtended)
“ANA to linear algebra” concrete implementation
support subclasses(e.g. MPIVectorBase)
(currently in TSFCore)
Interfaces to steady-state nonlinear applications
(currently in TSFCore::Nonlin,similar to NOX Group)
Interfaces to iterative linear solvers
(currently in Belos?)
Interfaces to transient nonlinear applications
(currently do not exist?)
Various extended “strategy” interfaces
(???)
???
“ANA to linear algebra” Interfaces
(e.g. Vector, LinearOp)(currently in TSFCore)
Extended APP-LAL and LAL-LAL interfaces to linear algebra objects(e.g. LoadableVector)
(currently in TSFExtended)
Extended APP-LAL and LAL-LAL handles
(currently in TSFExtended)
???
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Interfacing Packages to Implementations : Everyone for themselves!
Package 1
Package Interface 1
Package Interface 1 / Implementation 1
adapter
…
Implementation 1 Implementation 2
Package Interface 1 / Implementation 2
adapter
Package Interface 1 / Implementation N
adapter
Implementation N…
Package 2
Package Interface 2
Package Interface 2 / Implementation 1
adapter
Package Interface 2 / Implementation 2
adapter
Package Interface 2 / Implementation N
adapter
Package M
Package Interface M
Package Interface M / Implementation 1
adpater
Package Interface M / Implementation 2
adapter
Package Interface M / Implementation N
adapter
… … …
- Oh no, M * N adapter subclasses needed!- This is not a “scalable” approach!Package == ANA
Interfacing Packages to Implementations : Standard Interfaces (I)
Package 1
Package Interface 1
…
Implementation 1 Implementation 2 Implementation N…
Package 2
Package Interface 2
Package M
Package Interface M
+ Only M + N adapter subclasses needed!+ This is a “scalable” approach!
Std LANA Interface
Std LANA Interface / Implementation 1
adapter
Std LANA Interface / Implementation 2
adapter
Std LANA Interface / Implementation N
adapter
Package Interface 1 / Std LANA Interface
adapter
Package Interface 2 / Std LANA Interface
adapter
Package Interface M / Std LANA Interface
adapter
…
Interfacing Packages to Implementations : Standard Interfaces (II)
Package 1 …
Implementation 1 Implementation 2 Implementation N…
Package 2 Package M
+ Only N adapter subclasses needed!+ Less interface documentation, shared testing code
Std LANA Interface
Std LANA Interface / Implementation 1
adapter
Std LANA Interface / Implementation 2
adapter
Std LANA Interface / Implementation N
adapter
…
Interfacing Packages to Packages : Everyone for themselves!
Package 1
Package Interface 1
…
Package 2
Package Interface 2
Package Interface 2 / Package Interface 1
adapter
Package 3
Package Interface 3
Package Interface 3 / Package Interface 2
adapter
Package Interface 3 / Package Interface 1
adapter
- Major assumption: “Package Interface 1” satisfies the requirements of “Package Interface 2”?
Package M
- Oh no, as many as 1 + 2 + 3 + … + M-1 = O(M2) adapter subclasses needed if only one-way interactions!
Concrete Examples:
1) NOX/LOCA interfaces had to be extended to support Belos interfaces
2) NOX/LOCA vector interface does not support MOOCHO vector interface
Package Interface M
…
- This is not a “scalable” approach!
Interfacing Packages to Packages : Standard Interfaces (I)
Package Interface 1
…Package 2
Package Interface 2
Implementation 1
Std LANA Interface
Std LANA Interface / Implementation 1
adapter
Package Interface 1 / Std LANA Interface
adapter
Implementation 2
Std LANA Interface / Implementation 2
adapter
Extended LANA Interface 1
Package Interface 2 / Std LANA Interface
adapter
Package Interface 1 / Std LANA
Interface adapter
…
+ Only M adapters need+ Each Package knows nothing of other packages
- Result: Destroys ability to dynamic cast to “Extended LANA Inteface1” !
Package M
- Major assumption!!!“Package Interface 1” supports functionality of “Std LANA Interface”?
Concrete Example:
Belos::BlockGmres using Epetra adapters defines a TSFCore::LinearOp subclass to be used as a preconditioner in another package that also uses Epetra adapters.
Package 1
Concrete Example Problem : Belos, Epetra and TSFCore
TSFCore::LinearOp
TSFCore::MultiVector
TSFCore::VectorSpace
Belos::Operator
TSFCore::EpetraLinearOp
TSFCore::MPIVectorSpaceBase
Belos::TSFCoreOperator
Belos::TSFCoreMultiVec
TSFCore::MPIMultiVectorBase
Belos::BlockGmres
<<create>>
The problem:• TSFCore::BelosLinearOp::range() returns a TSFCore::BelosVectorSpace object• The TSFCore::BelosVectorSpace object creates a TSFCore::BelosMultiVector object• MPIMultiVectorBase::applyOp( … ) is called on EpetraMultiVector object passing in the TSFCore::BelosMultiVector object• TSFCore::BelosMultiVector::range() is called which returns a TSFCore::BelosVectorSpace object• The dynamic cast from TSFCore::BelosVectorSpace to TSFCore::MPIVectorSpaceBase fails
TSFCore::BelosMultiVectorTSFCore::BelosVectorSpace
range, domainrange, domain
The scenario: Create a MultiVector object use the range space of aTSFCore::BelosLinearOp object and assign it to a MultiVector object created from a TSFCore::EpetraVectorSpace.
range, domainTSFCore::EpetraVectorSpace
TSFCore::EpetraMultiVector
<<create>>
Belos::MultiVec
TSFCore::BelosLinearOp
The Delema: What to use for range and domain spaces?
Define TSFCore::VectorSpace only in terms of Belos interfaces!
Concrete Example Solution : Belos, Epetra and TSFCore
TSFCore::LinearOp
TSFCore::MultiVector
TSFCore::VectorSpace
Belos::Operator
TSFCore::EpetraLinearOp
TSFCore::MPIVectorSpaceBase
Belos::TSFCoreOperator
Belos::TSFCoreMultiVec
TSFCore::MPIMultiVectorBase
Belos::BlockGmres
The solution:• TSFCore::BelosLinearOp::range() returns a TSFCore::EpetraVectorSpace object• The TSFCore::EpetraVectorSpace object creates a TSFCore::EpetraMultiVector object• MPIMultiVectorBase::applyOp( … ) is called on EpetraMultiVector object passing in the TSFCore::EpetraMultiVector object• TSFCore::EpetraMultiVector::range() is called which returns a TSFCore::EpetraVectorSpace object• The dynamic cast from TSFCore::EpetraVectorSpace to TSFCore::MPIVectorSpaceBase succeeds
The scenario: Create a MultiVector object use the range space of aTSFCore::BelosLinearOp object and assign it to a MultiVector object created from a TSFCore::EpetraVectorSpace.
range, domainTSFCore::EpetraVectorSpace
TSFCore::EpetraMultiVector
<<create>>
Belos::MultiVec
The Delema: What to use for range and domain spaces?
Define TSFCore::VectorSpace in terms of underlying TSFCore::VectorSpace!
TSFCore::BelosLinearOp
Interfacing Packages to Packages : Standard Interfaces (II)
Package Interface 1
…Package 2
Package Interface 2
Implementation 1
Std LANA Interface
Std LANA Interface / Implementation 1
adapter
Package Interface 1 / Std LANA Interface
adapter
Implementation 2
Std LANA Interface / Implementation 2
adapter
Extended LANA Interface 1
Package Interface 2 / Std LANA Interface
adapter
…
+ Fewer new adapters needed
+ Dynamic casting of most objects maintained
Package Interface 1 / Std LANA
Interface adapter
The Recommended Practice!
Package 1
Pros and Cons of Standard Interfaces
Reasons to adopt “standard” interfaces
• “Automatic” interoperability only comes through “standard” interfaces
• Only way to guarantee interoperability is even possible (without revisions)
• One set of documentation
• Common set of unit tests for common interface objects
Reasons not to adopt “standard” interfaces
• Greater package autonomy / fewer dependencies
• Fewer layers of indirection within a single package
• Less sophisticated object-oriented concepts within a single package
Outline
• Introduction of abstract numerical algorithms (ANAs)
• Background on Unified Modeling Language (UML) and “Design Patterns”
• Overview of TSF interfaces and software
• Discussion of different approaches for interoperability between Trilinos packages
• General recommendations and open questions related to TSF and Trilinos
Recommendations for TSF and Trilinos
Goal: Adopt “standard interfaces” but minimize dependencies
• Minimize “standard” interface code needed for interoperability
• Encourage use of “standard” interfaces but allow packages to keep their own interfaces as local APIs
• All ASC/internal uses of Trilinos packages should adhere to “standard interfaces”
• Encourage academic and external collaborators to use “standard interfaces” but offer local package interfaces as an alternative
• Partition “standard” interface code as much as possible to minimize dependencies:
• Linear ANA interfaces (currently in TSFCore)
• SPMD Matrix/Vector loading and accessing (currently in TSFExtended)
• Interfaces to linear solvers (being developed in Belos)
• Interfaces to steady-state nonlinear problems (currently in TSFCore::Nonlin)
• Interfaces to transient non-linear problems (not yet developed)
• Separate “convenience” code from “standard interface” code
• Composite LANA objects (i.e. product vectors) (currently in TSFCore)
• Composite APP/LAL objects (currently in TSFCore and TSFExtended)
• Handle classes and operator overloading (currently in TSFExtended)
Open Questions related to TSF
• How do we make it “easiest” to develop concrete TSF interface implementations of vector spaces, vectors, multi-vectors and linear operators?
• Is it reasonable to expect beginner C++ programmers to debug though all TSF code?
Analogy: Should beginning Microsoft Windows users be able to debug though Windows code when something goes wrong?
Example: STL is good but difficult to impossible to debug through
• With that said, is the code in TSFCore and TSFExtended too complicated for semi-experienced C++ developers to debug through?
• What type of documentation for TSF is required beyond what is already available?
• How should the code in TSF be organized in to packages and managed in packages?
• Should all linear algebra interfaces be put into a single package?
• Should handles for operator overloading automatically include matrix/vector loading and extraction interfaces?
• Should certain types of code be allowed to be disabled at configure time?
TSFCore Support for APP-specific Scalar Products
Linear operators must “obey” application-specific scalar product <x,y>
Specifically, for the linear operator (i.e. LinearOp, MultiVector etc.):
the following adjoint relation must hold:
Goal of TSFCore support subclasses
=> Separate the definition of application-specific scalar product from data-structure and computing platform concrete implementations of vector spaces, vectors, multi-vectors and linear operators as much as possible.
VectorSpace LinearOp
ScalarProd
EuclideanScalarProd LinearOpScalarProd
rangeScalarProdVecSpc, domainScalarProdVecSpc
ScalarProdVectorSpaceBaseEuclideanLinearOpBase
op
AppSpecificScalarProd…
ConcreteMultiVector …
ConcreteVectorSpace …
Different Approaches for Developing Concrete Subclasses
Approach 1: Define “simpler” interfaces and include standard “adapter” subclasses• Example: NOX::Abstract::Group, NOX::Epetra::Group and NOX::Epetra::InterfaceApproach 2: Define templated adapter subclasses with “simpler” traits classes• Example: fei::Vector, snl_fei::Vector<> and snl_fei::VectorTraits<>Approach 3: Define support base classes that override “general” functions in terms of “simpler” pure
virtual functions which adapter subclasses define• Currently used by TSFCore => Example: TSFCore::SerialVectorBase, TSFCore::MPIVectorBase• Advantages
• Provides tailored “simple” set of functions to override• Subclasses have full control to override every virtual function, not just “simpler” functions
• Disadvantages• Subclass developers will see the “general” functions even though they should, in most cases,
ignore themGeneral Vector
…
Serial Vector Implementation 1
General Serial Vector Base
Serial Vector Implementation 2
MPI Vector Implementation 1
MPI Vector Implementation 2
General MPI Vector Base
Serial Vector Implementation 1
Adapter
Serial Vector Implementation 2
Adapter
MPI Vector Implementation 1
Adapter
MPI Vector Implementation 2
Adapter
…
…
……
Different Approaches for Developing Concrete Subclasses
Approach 1: Define “simpler” interfaces and include standard “adapter” subclasses• Example: NOX::Abstract::Group, NOX::Epetra::Group and NOX::Epetra::Interface• Advantages
• Provides tailored “simple” interfaces for subclass development for specific categories of implementations
• Subclass developers never even see “general” interface• Disadvantages
• Only operations in “simple” interface can be overridden, not in “general” interface.
General Vector
…
Serial Vector Implementation 1
General Serial Vector Adapter
Serial Vector Implementation 2
MPI Vector Implementation 1
MPI Vector Implementation 2
General MPI Vector Adapter
Simple Serial Vector Base
Simple MPI Vector Base
Serial Vector Implementation 1
Adapter
Serial Vector Implementation 2
Adapter
MPI Vector Implementation 1
Adapter
MPI Vector Implementation 2
Adapter
…
…
……
Different Approaches for Developing Concrete Subclasses
Approach 2: Define templated adapter subclasses with “simpler” traits classes• Example: fei::Vector, snl_fei::Vector<> and snl_fei::VectorTraits<>• Advantages
• Provides tailored “simple” traits interfaces and does not require writing subclasses• Less subclasses than Approach 1• Subclass developers never even see “general” interface
• Disadvantages• Only operations in “simple” traits interface can be overridden, not in “general” interface • Templated traits may or may not be more complicated than using general inheritance• Traits can not maintain state for a particular object while general subclasses can
General Vector
Serial Vector Implementation 1
General Serial Vector Adapter
Serial Vector Implementation 2
MPI Vector Implementation 1
MPI Vector Implementation 2
General MPI Vector Adapter
…
……
ConcreteType ConcreteType
• Create traits class specializations for all concrete implementations• Instantiate templated adapters on the concrete implementation types
Uses SerialVectorTraits<> Uses MPIVectorTraits<>