numerical generic programming with c++ templates
TRANSCRIPT
Numerical Generic Programming with C++ Templates
Roy H. Stogner
The University of Texas at Austin
Jan 18, 2012
Roy H. Stogner Generic Jan 18, 2012 1 / 44
Generic Programming Data Types
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 2 / 44
Generic Programming Data Types
Primitive Data Types
Exampleconst float theta = 0.5;
double onethird = 1.0/3.0; // Not 1/3!
int power = 2;
int* ptr_to_power = &power;
bool goodexamples = false;
const char* nexttask = "do better";
• Data types native to the CPU
• Data addresses in memory
• Logical extensions, reinterpretations of those types
• “Constant” correctness checking
Roy H. Stogner Generic Jan 18, 2012 3 / 44
Generic Programming Data Types
Data Type Safety
Weakly Typed Languages
Example
a = 2;
b = "2";
concatenate(a, b) # "22"
add(a, b) # 4
Bash, Perl, etc.
Strongly Typed Languages
Example
int a = 2;
string b = "2";
concatenate(a, b); // error
add(a, b); // error
concatenate(string(a), b); // "22"
add(a, int(b)); // 4
C++, Fortran, Python, etc.
• But even C++ lets you write implicit conversion operators
• And even Bash screams at runtime if it can’t reinterpret data asrequested
Roy H. Stogner Generic Jan 18, 2012 4 / 44
Generic Programming Data Types
Data Type Safety
Weakly Typed Languages
Example
a = 2;
b = "2";
concatenate(a, b) # "22"
add(a, b) # 4
Bash, Perl, etc.
Strongly Typed Languages
Example
int a = 2;
string b = "2";
concatenate(a, b); // error
add(a, b); // error
concatenate(string(a), b); // "22"
add(a, int(b)); // 4
C++, Fortran, Python, etc.
• But even C++ lets you write implicit conversion operators
• And even Bash screams at runtime if it can’t reinterpret data asrequested
Roy H. Stogner Generic Jan 18, 2012 4 / 44
Generic Programming Data Types
Data Type Safety
Weakly Typed Languages
Example
a = 2;
b = "2";
concatenate(a, b) # "22"
add(a, b) # 4
Bash, Perl, etc.
Strongly Typed Languages
Example
int a = 2;
string b = "2";
concatenate(a, b); // error
add(a, b); // error
concatenate(string(a), b); // "22"
add(a, int(b)); // 4
C++, Fortran, Python, etc.
• But even C++ lets you write implicit conversion operators
• And even Bash screams at runtime if it can’t reinterpret data asrequested
Roy H. Stogner Generic Jan 18, 2012 4 / 44
Generic Programming Data Types
Structured Types
Example
class ComplexFloat {
ComplexFloat(float re, float im);
ComplexFloat(float re);
typedef float value_type;
.........
float norm() const {
return sqrt(realpart*realpart +
imagpart*imagpart);
}
private:
float realpart, imagpart;
static const bool is_complex = true;
};
• DataI Private?I Static?
• MethodsI ConstructorsI DestructorsI AccessorsI Inline?
• Subtype DefinitionsI Way more powerful
than they look...
Roy H. Stogner Generic Jan 18, 2012 5 / 44
Generic Programming Data Types
Structured Types
Example
class ComplexFloat {
ComplexFloat(float re, float im);
ComplexFloat(float re);
typedef float value_type;
.........float norm() const {
return sqrt(realpart*realpart +
imagpart*imagpart);
}
private:
float realpart, imagpart;
static const bool is_complex = true;
};
• DataI Private?I Static?
• MethodsI ConstructorsI DestructorsI AccessorsI Inline?
• Subtype DefinitionsI Way more powerful
than they look...
Roy H. Stogner Generic Jan 18, 2012 5 / 44
Generic Programming Data Types
Structured Types
Example
class ComplexFloat {
ComplexFloat(float re, float im);
ComplexFloat(float re);
typedef float value_type;.........float norm() const {
return sqrt(realpart*realpart +
imagpart*imagpart);
}
private:
float realpart, imagpart;
static const bool is_complex = true;
};
• DataI Private?I Static?
• MethodsI ConstructorsI DestructorsI AccessorsI Inline?
• Subtype DefinitionsI Way more powerful
than they look...
Roy H. Stogner Generic Jan 18, 2012 5 / 44
Generic Programming Data Types
Overloading
Example
inline ComplexFloat operator*(const ComplexFloat& a,
const ComplexFloat& b) {
return ComplexFloat(a.realpart*b.realpart-a.imagpart*b.imagpart,
a.realpart*b.imagpart+a.imagpart*b.realpart);
}
const ComplexFloat i(0,1), three(3,0);
ComplexFloat old_syntax_3i, function_overloaded_3i;
old_syntax_3i.equals(ComplexFloat_times(three,i));
function_overloaded_3i.equals(times(three,i));
const ComplexFloat new_3i = three * i;
• To computer scientists: overloading is “syntactic sugar”, which justlets you write the exact same program a little less bitterly.
• To engineers: overloading lets your math look like math!
Roy H. Stogner Generic Jan 18, 2012 6 / 44
Generic Programming Data Types
Overloading
Example
inline ComplexFloat operator*(const ComplexFloat& a,
const ComplexFloat& b) {
return ComplexFloat(a.realpart*b.realpart-a.imagpart*b.imagpart,
a.realpart*b.imagpart+a.imagpart*b.realpart);
}
const ComplexFloat i(0,1), three(3,0);
ComplexFloat old_syntax_3i, function_overloaded_3i;
old_syntax_3i.equals(ComplexFloat_times(three,i));
function_overloaded_3i.equals(times(three,i));
const ComplexFloat new_3i = three * i;
• To computer scientists: overloading is “syntactic sugar”, which justlets you write the exact same program a little less bitterly.
• To engineers: overloading lets your math look like math!
Roy H. Stogner Generic Jan 18, 2012 6 / 44
Generic Programming Data Types
Overloading
Example
inline ComplexFloat operator*(const ComplexFloat& a,
const ComplexFloat& b) {
return ComplexFloat(a.realpart*b.realpart-a.imagpart*b.imagpart,
a.realpart*b.imagpart+a.imagpart*b.realpart);
}
const ComplexFloat i(0,1), three(3,0);
ComplexFloat old_syntax_3i, function_overloaded_3i;
old_syntax_3i.equals(ComplexFloat_times(three,i));
function_overloaded_3i.equals(times(three,i));
const ComplexFloat new_3i = three * i;
• To computer scientists: overloading is “syntactic sugar”, which justlets you write the exact same program a little less bitterly.
• To engineers: overloading lets your math look like math!
Roy H. Stogner Generic Jan 18, 2012 6 / 44
Generic Programming Data Types
Data Type Dependence
Example
class VectorFloat {.........float dot(const VectorFloat& v) const {
float r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........float* data;
};
class VectorDouble {.........class VectorComplexDouble {.........class VectorTendonitis {.........
But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!
Roy H. Stogner Generic Jan 18, 2012 7 / 44
Generic Programming Data Types
Data Type Dependence
Example
class VectorFloat {.........float dot(const VectorFloat& v) const {
float r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........float* data;
};
class VectorDouble {.........
class VectorComplexDouble {.........class VectorTendonitis {.........
But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!
Roy H. Stogner Generic Jan 18, 2012 7 / 44
Generic Programming Data Types
Data Type Dependence
Example
class VectorFloat {.........float dot(const VectorFloat& v) const {
float r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........float* data;
};
class VectorDouble {.........class VectorComplexDouble {.........
class VectorTendonitis {.........
But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!
Roy H. Stogner Generic Jan 18, 2012 7 / 44
Generic Programming Data Types
Data Type Dependence
Example
class VectorFloat {.........float dot(const VectorFloat& v) const {
float r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........float* data;
};
class VectorDouble {.........class VectorComplexDouble {.........class VectorTendonitis {.........
But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!
Roy H. Stogner Generic Jan 18, 2012 7 / 44
Generic Programming Data Types
Data Type Dependence
Example
class VectorFloat {.........float dot(const VectorFloat& v) const {
float r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........float* data;
};
class VectorDouble {.........class VectorComplexDouble {.........class VectorTendonitis {.........
But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!
Roy H. Stogner Generic Jan 18, 2012 7 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........virtual Number&
operator+=(const Number& n)
= 0; // Pure virtual.........};
• Create an abstract interface
• Derive concrete subclassesfrom it
• Implement the interface’sfunctions in each subclass
• Hold pointers or references tothe base class
• Write code to use the interface
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........virtual Number&
operator+=(const Number& n)
= 0; // Pure virtual.........};
class FloatNumber :
public Number {.........};
class DoubleNumber :
public Number {.........};
• Create an abstract interface
• Derive concrete subclassesfrom it
• Implement the interface’sfunctions in each subclass
• Hold pointers or references tothe base class
• Write code to use the interface
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........virtual Number&
operator+=(const Number& n)
= 0; // Pure virtual.........};
class FloatNumber :
public Number {.........virtual Number&
operator+=(const Number& n) {
datum += n.data();
return *this;
}.........float datum;.........};
• Create an abstract interface
• Derive concrete subclassesfrom it
• Implement the interface’sfunctions in each subclass
• Hold pointers or references tothe base class
• Write code to use the interface
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented ProgrammingExample
class Number {.........};
class FloatNumber :
public Number {.........virtual Number&
operator+=(const Number& n) {
datum += n.data();
return *this;
}.........float datum;.........};
class VectorNumber {.........
Number** data;
};
• Create an abstract interface
• Derive concrete subclassesfrom it
• Implement the interface’sfunctions in each subclass
• Hold pointers or references tothe base class
• Write code to use the interface
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Create an abstract interface
• Derive concrete subclassesfrom it
• Implement the interface’sfunctions in each subclass
• Hold pointers or references tothe base class
• Write code to use the interface
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:
I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctly
I Defines concept relations• Number is an Abstract Base
Class• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:
I Can require “shim” classes• A float isn’t a
FloatNumber!I Slow as hell
• Fragmented dynamicmemory
• Virtual function calls forevery FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Object Oriented Programming
Example
class Number {.........};
class FloatNumber :
public Number {.........};
class VectorNumber {.........Number dot(const VectorNumber& v)
const {
Number r = 0;
for(int i = 0; i != size; ++i)
r += (*data)[i] * v[i];
return r;
}.........Number** data;
};
• Advantages:I Works correctlyI Defines concept relations
• Number is an Abstract BaseClass
• FloatNumber is-a Number
I Heterogenous containers!• data[0] can be double,• data[1] can be complex...
• Disadvantages:I Can require “shim” classes
• A float isn’t aFloatNumber!
I Slow as hell• Fragmented dynamic
memory• Virtual function calls for
every FLOP!
Roy H. Stogner Generic Jan 18, 2012 8 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T* data;
};
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• Template arguments can bedata types!
• Class data members candepend on template argument
• Class methods can depend ontemplate argument
• Templated classes can beinstantiated with plain datatypes
I ... or with other instantiatedtemplated types
I ... or with other instantiationsof themselves!
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• Advantages
I Works with existing typesI Functions can be inlined, not
just virtualI Intermediate calculation types
can be implicit• Disadvantages
I Makes assumptions aboutinput types
• Is that NumericVectoroperator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• AdvantagesI Works with existing types
I Functions can be inlined, notjust virtual
I Intermediate calculation typescan be implicit
• DisadvantagesI Makes assumptions about
input types• Is that NumericVector
operator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• AdvantagesI Works with existing typesI Functions can be inlined, not
just virtual
I Intermediate calculation typescan be implicit
• DisadvantagesI Makes assumptions about
input types• Is that NumericVector
operator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• AdvantagesI Works with existing typesI Functions can be inlined, not
just virtualI Intermediate calculation types
can be implicit
• DisadvantagesI Makes assumptions about
input types• Is that NumericVector
operator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• AdvantagesI Works with existing typesI Functions can be inlined, not
just virtualI Intermediate calculation types
can be implicit• Disadvantages
I Makes assumptions aboutinput types
• Is that NumericVectoroperator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Data Type Independence: Generic Programming
Example
template <typename T>
class NumericVector {.........T dot(const NumericVector<T>& v)
const {
T r = 0;
for(int i = 0; i != size; ++i)
r += data[i] * v[i];
return r;.........T* data;
};
NumericVector<float>
floatvec;
NumericVector<complex<double> >
complexdoublevec;
NumericVector<NumericVector<float> >
floatmatrix;
• AdvantagesI Works with existing typesI Functions can be inlined, not
just virtualI Intermediate calculation types
can be implicit• Disadvantages
I Makes assumptions aboutinput types
• Is that NumericVectoroperator* an element byelement multiplication or adot product?
I Functions are undefined untilinstantiation
• Compile-time bugs goundetected until triggered
Roy H. Stogner Generic Jan 18, 2012 9 / 44
Generic Programming Data Types
Duck Typing
So we can duct tape our templated type to any type T that behaves like weexpect math to behave? Is that safe?
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We’ll bump into some of thoselater.
Roy H. Stogner Generic Jan 18, 2012 10 / 44
Generic Programming Data Types
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We’ll bump into some of thoselater.
Roy H. Stogner Generic Jan 18, 2012 10 / 44
Generic Programming Data Types
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We’ll bump into some of thoselater.
Roy H. Stogner Generic Jan 18, 2012 10 / 44
Generic Programming Data Types
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We’ll bump into some of thoselater.
Roy H. Stogner Generic Jan 18, 2012 10 / 44
Generic Programming Multi-precision verification
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 11 / 44
Generic Programming Multi-precision verification
Operator-Overloaded Multi-precision
Example
template <typename T, typename S>
struct ShadowNumber {.........T _val;
S _shadow;
};
• Simultaneous calculation with multiple floating point representations,to estimate rounding error:
ExampleShadowNumber<float, double> shadowed_float;
ShadowNumber<double, long double> shadowed_double;
Roy H. Stogner Generic Jan 18, 2012 12 / 44
Generic Programming Multi-precision verification
Operator-Overloaded Multi-precisionExample
template <typename T, typename S>
struct ShadowNumber {.........
template <typename T2>
ShadowNumber(const T2& val) : _val(val), _shadow(val) {}
template <typename T2, typename S2>
ShadowNumber(ShadowNumber<T2, S2>& other) :
_val(other._val), _shadow(other._shadow) {}.........T _val;
S _shadow;
};
• Functions can be templated, and templates can be nested, too!
ExampleShadowNumber<float, double> shadowed_float;
ShadowNumber<double, long double> shadowed_double;
Roy H. Stogner Generic Jan 18, 2012 12 / 44
Generic Programming Multi-precision verification
Operator-Overloaded Multi-precisionExample
template <typename T, typename S>
struct ShadowNumber {.........
template <typename T2>
ShadowNumber(const T2& val) : _val(val), _shadow(val) {}
template <typename T2, typename S2>
ShadowNumber(ShadowNumber<T2, S2>& other) :
_val(other._val), _shadow(other._shadow) {}.........
template <typename T2>
ShadowNumber<T,S>& operator+= (const T2& a)
{ _val += a; _shadow += a; return *this; }
template <typename T2, typename S2>
ShadowNumber<T,S>& operator+= (const ShadowNumber<T2,S2>& a)
{ _val += a.value(); _shadow += a.shadow(); return *this; }.........T _val;
S _shadow;
};
•
ExampleShadowNumber<float, double> shadowed_float;
ShadowNumber<double, long double> shadowed_double;
Roy H. Stogner Generic Jan 18, 2012 12 / 44
Generic Programming Array Operations
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 13 / 44
Generic Programming Array Operations
Operator-Overloaded Vectors
Example
template <unsigned int size, typename T>
• Class templates can have integer (long, short, bool, etc.) arguments,not just data type arguments
• This lets us fix array sizes at compile time - no heap memory
• Fixed length loops can unroll with no branching
Roy H. Stogner Generic Jan 18, 2012 14 / 44
Generic Programming Array Operations
Operator-Overloaded Vectors
Example
template <unsigned int size, typename T>
struct NumberArray {.........
T _data[size];
};
• Class templates can have integer (long, short, bool, etc.) arguments,not just data type arguments
• This lets us fix array sizes at compile time - no heap memory
• Fixed length loops can unroll with no branching
Roy H. Stogner Generic Jan 18, 2012 14 / 44
Generic Programming Array Operations
Operator-Overloaded Vectors
Example
template <unsigned int size, typename T>
struct NumberArray {.........
template <typename T2>
NumberArray<size,T>& operator+= (const NumberArray<size,T2>& a) {
for (unsigned int i = 0; i != size; ++i)
_data[i] += a[i]; return *this;
}.........T _data[size];
};
• Class templates can have integer (long, short, bool, etc.) arguments,not just data type arguments
• This lets us fix array sizes at compile time - no heap memory
• Fixed length loops can unroll with no branching
Roy H. Stogner Generic Jan 18, 2012 14 / 44
Generic Programming Array Operations
Element-by-element Vector Functions
Example
#define NumberArray_std_unary(funcname) \
template <unsigned int size, typename T> \
inline \
NumberArray<size, T> \
funcname (const NumberArray<size, T>& a) \
{ \
NumberArray<size, T> returnval; \
\
for (unsigned int i = 0; i != size; ++i) \
returnval[i] = funcname(a[i]); \
\
return returnval; \
}
NumberArray_std_unary(exp)
NumberArray_std_unary(sin)
NumberArray_std_unary(cos)
• Templated function andstruct definitions can’t beinstantiated with thename to be defined as atemplate argument.
• We instantiate templatesto avoid writing NTNf
repetitive algorithms withunsafe macros
• We still need unsafemacros to avoid writingNf repetitive templateinstantiations.
Roy H. Stogner Generic Jan 18, 2012 15 / 44
Generic Programming Array Operations
Element-by-element Vector Functions
Example
#define NumberArray_std_unary(funcname) \
template <unsigned int size, typename T> \
inline \
NumberArray<size, T> \
funcname (const NumberArray<size, T>& a) \
{ \
NumberArray<size, T> returnval; \
\
for (unsigned int i = 0; i != size; ++i) \
returnval[i] = funcname(a[i]); \
\
return returnval; \
}
NumberArray_std_unary(exp)
NumberArray_std_unary(sin)
NumberArray_std_unary(cos)
• Templated function andstruct definitions can’t beinstantiated with thename to be defined as atemplate argument.
• We instantiate templatesto avoid writing NTNf
repetitive algorithms withunsafe macros
• We still need unsafemacros to avoid writingNf repetitive templateinstantiations.
Roy H. Stogner Generic Jan 18, 2012 15 / 44
Generic Programming Array Operations
Element-by-element Vector Functions
Example
#define NumberArray_std_unary(funcname) \
template <unsigned int size, typename T> \
inline \
NumberArray<size, T> \
funcname (const NumberArray<size, T>& a) \
{ \
NumberArray<size, T> returnval; \
\
for (unsigned int i = 0; i != size; ++i) \
returnval[i] = funcname(a[i]); \
\
return returnval; \
}
NumberArray_std_unary(exp)
NumberArray_std_unary(sin)
NumberArray_std_unary(cos)
• Templated function andstruct definitions can’t beinstantiated with thename to be defined as atemplate argument.
• We instantiate templatesto avoid writing NTNf
repetitive algorithms withunsafe macros
• We still need unsafemacros to avoid writingNf repetitive templateinstantiations.
Roy H. Stogner Generic Jan 18, 2012 15 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f);
// Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v);
// Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f);
// Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Array Operations
Operator-Overloaded VectorsExample
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Uh-oh!
What is the return type supposed to be for
template <unsigned int size1, typename T1,
unsigned int size2, typename T2>
operator+(const NumberArray<size1,T1>& a,
const NumberArray<size2,T2>& b)
???
Roy H. Stogner Generic Jan 18, 2012 16 / 44
Generic Programming Computation with Data Types
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 17 / 44
Generic Programming Computation with Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
And then we can specialize!
Exampletemplate <>
struct CompareTypes<float,double> {
typedef double
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 18 / 44
Generic Programming Computation with Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
Or, for class templates, partially specialize!
Exampletemplate <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,typename CompareTypes<T,T2>::supertype>
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 18 / 44
Generic Programming Computation with Data Types
Interlude: Functional Programming
• Wait, we can partially specialize class templates?
• Oops, we just created a new Turing-complete metalanguage.
• Anything computable with integers is now computable at compile time!
• ...with atrocious syntax...
Roy H. Stogner Generic Jan 18, 2012 19 / 44
Generic Programming Computation with Data Types
Interlude: Functional Programming
• Wait, we can partially specialize class templates?
• Oops, we just created a new Turing-complete metalanguage.
• Anything computable with integers is now computable at compile time!
• ...with atrocious syntax...
Roy H. Stogner Generic Jan 18, 2012 19 / 44
Generic Programming Computation with Data Types
Interlude: Functional Programming
• Wait, we can partially specialize class templates?
• Oops, we just created a new Turing-complete metalanguage.
• Anything computable with integers is now computable at compile time!
• ...with atrocious syntax...
Roy H. Stogner Generic Jan 18, 2012 19 / 44
Generic Programming Computation with Data Types
Interlude: Functional Programming
• Wait, we can partially specialize class templates?
• Oops, we just created a new Turing-complete metalanguage.
• Anything computable with integers is now computable at compile time!
• ...with atrocious syntax...
Roy H. Stogner Generic Jan 18, 2012 19 / 44
Generic Programming Computation with Data Types
Functional Programming
Imperative, run-time C++ calculation of N !:
Exampleunsigned int factorial(unsigned int N) {
unsigned int f = 1;
for (unsigned int i = 2; i <= N; ++i)
f *= i; Value is variable
return f;
}
// Three FP ops, four test-and-branches
unsigned int twelve = factorial(4);
Roy H. Stogner Generic Jan 18, 2012 20 / 44
Generic Programming Computation with Data Types
Functional Programming
Declarative, compile-time C++ calculation of N !:
Exampletemplate<unsigned int N>
struct Factorial {
static const unsigned int value = Each value
N * Factorial<N-1>::value; is constant
};
template<>
struct Factorial<0> {
static const unsigned int value = 1;
};
// Zero FP ops, zero tests/branches at runtime
unsigned int twelve = Factorial<4>::value;
Roy H. Stogner Generic Jan 18, 2012 20 / 44
Generic Programming Computation with Data Types
Back to Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,double
>::supertype) ==
typeid(NumberArray<3,double>))
how_much_we_rock = 10;
Roy H. Stogner Generic Jan 18, 2012 21 / 44
Generic Programming Computation with Data Types
Back to Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,NumberArray<float> >,double
>::supertype) ==
typeid(NumberArray<3,NumberArray<double> >))
how_much_we_rock = 11; // That extra push over the cliff
Roy H. Stogner Generic Jan 18, 2012 21 / 44
Generic Programming Computation with Data Types
Back to Return Types: Here Be Dragons
A ShadowNumber of something, combined with anything else we haven’tseen before
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
ShadowNumber<float,double>,double
>::supertype) ==
typeid(ShadowNumber<double,double>))
how_much_we_rock = 7; // C is still a passing grade
Roy H. Stogner Generic Jan 18, 2012 21 / 44
Generic Programming Computation with Data Types
Back to Return Types: Here Be Dragons
A ShadowNumber of something, combined with anything else we haven’tseen before
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(NumberArray<3,ShadowNumber<float,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Generic Jan 18, 2012 21 / 44
Generic Programming Computation with Data Types
Back to Return Types: Here Be Dragons
A ShadowNumber of something, combined with anything else we haven’tseen before
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(ShadowNumber<NumberArray<3,float>,NumberArray<3,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Generic Jan 18, 2012 21 / 44
Generic Programming Computation with Data Types
Back to Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization todisambiguate every combination
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 22 / 44
Generic Programming Computation with Data Types
Back to Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization todisambiguate every combination
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 22 / 44
Generic Programming Computation with Data Types
Back to Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization todisambiguate every combination
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 22 / 44
Generic Programming Computation with Data Types
Back to Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization todisambiguate every combination
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Generic Jan 18, 2012 22 / 44
Generic Programming Computation with Data Types
Return Type Magic: SFINAE
Example
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<NumberArray<size,T1>,T2>::supertype
operator+(const NumberArray<size1,T1>& a,
const T2& b)
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<ShadowNumber<T2,S2>,T1>::supertype
operator+(const T1& a,
const ShadowNumber<T2,S2>& b)
• Substitution Failure Is Not An Error
• These ambiguous functions are no longer ambiguous!
Roy H. Stogner Generic Jan 18, 2012 23 / 44
Generic Programming Computation with Data Types
Return Type Magic: SFINAE
Example
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<NumberArray<size,T1>,T2>::supertype
operator+(const NumberArray<size1,T1>& a,
const T2& b)
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<ShadowNumber<T2,S2>,T1>::supertype
operator+(const T1& a,
const ShadowNumber<T2,S2>& b)
• Substitution Failure Is Not An Error
• These ambiguous functions are no longer ambiguous!
Roy H. Stogner Generic Jan 18, 2012 23 / 44
Generic Programming Computation with Data Types
Return Type Magic: SFINAE
Example
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<NumberArray<size,T1>,T2>::supertype
operator+(const NumberArray<size1,T1>& a,
const T2& b)
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<ShadowNumber<T2,S2>,T1>::supertype
operator+(const T1& a,
const ShadowNumber<T2,S2>& b)
• Substitution Failure Is Not An Error
• These ambiguous functions are no longer ambiguous!
Roy H. Stogner Generic Jan 18, 2012 23 / 44
Generic Programming Computation with Data Types
Return Type Magic: SFINAE
Example
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<NumberArray<size,T1>,T2>::supertype
operator+(const NumberArray<size1,T1>& a,
const T2& b)
template <unsigned int size1, typename T1,
typename T2>
typename CompareTypes<ShadowNumber<T2,S2>,T1>::supertype
operator+(const T1& a,
const ShadowNumber<T2,S2>& b)
• Substitution Failure Is Not An Error
• These ambiguous functions are no longer ambiguous!
Roy H. Stogner Generic Jan 18, 2012 23 / 44
Generic Programming Automatic Differentiation
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 24 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a single new element ε to R
• Close under addition and multiplication:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a single new element ε to R• Close under addition and multiplication:{
m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]• Add a single new element ε to R• Close under addition and multiplication:{
m∑i=0
aiεi : ai ∈ R,m <∞
}• Take the quotient with ε2 ≡ − 1
D[R] ≡ {a+ bε : a, b ∈ R}
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]• Add a single new element ε to R• Close under addition and multiplication:{
m∑i=0
aiεi : ai ∈ R,m <∞
}• Take the quotient with ε2 ≡ − 1 ...no, wait, that’s C
D[R] ≡ {a+ bε : a, b ∈ R}
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]• Add a single new element ε to R• Close under addition and multiplication:{
m∑i=0
aiεi : ai ∈ R,m <∞
}• Take the quotient with ε2 ≡ 0
D[R] ≡ {a+ bε : a, b ∈ R}
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]• Add a single new element ε to R• Close under addition and multiplication:{
m∑i=0
aiεi : ai ∈ R,m <∞
}• Take the quotient with ε2 ≡ 0
D[R] ≡ {a+ bε : a, b ∈ R}
• Arithmetic:
(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)
(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)
(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2
= ((ac) + (ad+ bc)ε)
Roy H. Stogner Generic Jan 18, 2012 25 / 44
Generic Programming Automatic Differentiation
“Dual Numbers”
For each simple operation s(x, y),
s(a+ bε, c+ dε) = s(a, c) +
(b∂s
∂x(a, c) + d
∂s
∂y(a, c)
)ε
Extending to division, algebraic functions, transcendental functions, etcalso gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
Composition of functions preserves these properties, so with independentvariable x ∈ R, we set X ≡ x+ ε and get:
f(X) = f(x) + f ′(x)ε
This time the isomorphism practically codes itself!
Roy H. Stogner Generic Jan 18, 2012 26 / 44
Generic Programming Automatic Differentiation
“Dual Numbers”
For each simple operation s(x, y),
s(a+ bε, c+ dε) = s(a, c) +
(b∂s
∂x(a, c) + d
∂s
∂y(a, c)
)ε
Extending to division, algebraic functions, transcendental functions, etcalso gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
Composition of functions preserves these properties, so with independentvariable x ∈ R, we set X ≡ x+ ε and get:
f(X) = f(x) + f ′(x)ε
This time the isomorphism practically codes itself!
Roy H. Stogner Generic Jan 18, 2012 26 / 44
Generic Programming Automatic Differentiation
“Dual Numbers”
For each simple operation s(x, y),
s(a+ bε, c+ dε) = s(a, c) +
(b∂s
∂x(a, c) + d
∂s
∂y(a, c)
)ε
Extending to division, algebraic functions, transcendental functions, etcalso gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
Composition of functions preserves these properties, so with independentvariable x ∈ R, we set X ≡ x+ ε and get:
f(X) = f(x) + f ′(x)ε
This time the isomorphism practically codes itself!
Roy H. Stogner Generic Jan 18, 2012 26 / 44
Generic Programming Automatic Differentiation
“Dual Numbers”
For each simple operation s(x, y),
s(a+ bε, c+ dε) = s(a, c) +
(b∂s
∂x(a, c) + d
∂s
∂y(a, c)
)ε
Extending to division, algebraic functions, transcendental functions, etcalso gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
Composition of functions preserves these properties, so with independentvariable x ∈ R, we set X ≡ x+ ε and get:
f(X) = f(x) + f ′(x)ε
This time the isomorphism practically codes itself!
Roy H. Stogner Generic Jan 18, 2012 26 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
template <typename T, typename D>
struct DualNumber {.........T _value;
D _deriv;
};
Roy H. Stogner Generic Jan 18, 2012 27 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
template <typename T, typename D>
struct DualNumber {.........
template <typename T2>
DualNumber(const T2& val);
template <typename T2, typename D2>
DualNumber(const DualNumber<D2, T2>& val);.........T _value;
D _deriv;
};
Roy H. Stogner Generic Jan 18, 2012 27 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward DifferentiationExample
template <typename T, typename D>
struct DualNumber {.........
template <typename T2>
DualNumber<T,S>& operator*= (const T2& a)
{ _value *= a; _deriv *= a; return *this; }
template <typename T2, typename D2>
DualNumber<T,S>& operator*= (const DualNumber<T2,D2>& a)
{
_value *= a.value();
_deriv *= a.value();
_deriv += _value * a.derivatives();
return *this;
}.........T _value;
D _deriv;
};
Roy H. Stogner Generic Jan 18, 2012 27 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions!
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!
Roy H. Stogner Generic Jan 18, 2012 28 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions!
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!
Roy H. Stogner Generic Jan 18, 2012 28 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions!
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!
Roy H. Stogner Generic Jan 18, 2012 28 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions!
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!
Roy H. Stogner Generic Jan 18, 2012 28 / 44
Generic Programming Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions!
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!
Roy H. Stogner Generic Jan 18, 2012 28 / 44
Generic Programming Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R
• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• Arithmetic:
(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =
((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)
(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =
((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =
(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 29 / 44
Generic Programming Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• Arithmetic:
(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =
((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)
(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =
((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =
(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 29 / 44
Generic Programming Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• Arithmetic:
(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =
((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)
(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =
((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =
(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 29 / 44
Generic Programming Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• Arithmetic:
(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =
((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)
(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =
((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =
(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 29 / 44
Generic Programming Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• Arithmetic:
(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =
((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)
(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =
((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =
(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 29 / 44
Generic Programming Automatic Differentiation
“Hyper-Dual Numbers”
For each simple operation s(x, y),
s(a+ bε1 + bε2 + cε1ε2, d+ eε1 + eε2 + fε1ε2) =
s(a, d) +
(b∂s
∂x(a, d) + e
∂s
∂y(a, d)
)(ε1 + ε2)+(
c∂s
∂x+ b2
∂2s
∂2x+ 2be
∂2s
∂x∂y+ f
∂s
∂y+ e2
∂2s
∂2y
)ε1ε2
Extending gives: for any function f(x),
f(a+bε1+bε2+cε1ε2) = f(a)+bf ′(a)ε1+bf′(a)ε2+(b2f ′′(a)+cf ′(a))ε1ε2
With independent x ∈ R, set X ≡ x+ ε1 + ε2 and get:
f(X) = f(x) + f ′(x)ε1 + f ′(x)ε2 + f ′′(x)ε1ε2
Roy H. Stogner Generic Jan 18, 2012 30 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• Recall
I D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}
I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives
• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}
I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}
I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f′(a+ bε1)ε2
I = f(a) + bf ′(a)ε1 + (c+ dε1)(f′(a) + bf ′′(a)ε1)ε2
I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2
I = f(a) + bf ′(a)ε1 + (c+ dε1)(f′(a) + bf ′′(a)ε1)ε2
I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2
I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]
• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Generic Jan 18, 2012 31 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Templates
Example
typedef DualNumber<RawType,RawType> FirstDerivType;
typedef DualNumber<FirstDerivType,FirstDerivType> SecondDerivType;
// Initializing independent vars would be ugly:
const FirstDerivType x1(pi/6,1);
const FirstDerivType one(1,0);
const SecondDerivType x(x1,one);
// But smart constructor partial specialization
// makes it independent of recursion level:
const SecondDerivType x(pi/6,1);
const FirstDerivType sinx = sin(x); // Calculations like normal
const FirstDerivType y = sinx*sinx;
const double raw_y = raw_value(y); // Recursive down-conversion
double deriv = raw_value(y.derivatives());
double deriv2 = raw_value(y.derivatives().derivatives());
Roy H. Stogner Generic Jan 18, 2012 32 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Templates
Example
typedef DualNumber<RawType,RawType> FirstDerivType;
typedef DualNumber<FirstDerivType,FirstDerivType> SecondDerivType;
// Initializing independent vars would be ugly:
const FirstDerivType x1(pi/6,1);
const FirstDerivType one(1,0);
const SecondDerivType x(x1,one);
// But smart constructor partial specialization
// makes it independent of recursion level:
const SecondDerivType x(pi/6,1);
const FirstDerivType sinx = sin(x); // Calculations like normal
const FirstDerivType y = sinx*sinx;
const double raw_y = raw_value(y); // Recursive down-conversion
double deriv = raw_value(y.derivatives());
double deriv2 = raw_value(y.derivatives().derivatives());
Roy H. Stogner Generic Jan 18, 2012 32 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Templates
Example
typedef DualNumber<RawType,RawType> FirstDerivType;
typedef DualNumber<FirstDerivType,FirstDerivType> SecondDerivType;
// Initializing independent vars would be ugly:
const FirstDerivType x1(pi/6,1);
const FirstDerivType one(1,0);
const SecondDerivType x(x1,one);
// But smart constructor partial specialization
// makes it independent of recursion level:
const SecondDerivType x(pi/6,1);
const FirstDerivType sinx = sin(x); // Calculations like normal
const FirstDerivType y = sinx*sinx;
const double raw_y = raw_value(y); // Recursive down-conversion
double deriv = raw_value(y.derivatives());
double deriv2 = raw_value(y.derivatives().derivatives());
Roy H. Stogner Generic Jan 18, 2012 32 / 44
Generic Programming Automatic Differentiation
Hyper-dual Numbers via Recursive Templates
Example
typedef DualNumber<RawType,RawType> FirstDerivType;
typedef DualNumber<FirstDerivType,FirstDerivType> SecondDerivType;
// Initializing independent vars would be ugly:
const FirstDerivType x1(pi/6,1);
const FirstDerivType one(1,0);
const SecondDerivType x(x1,one);
// But smart constructor partial specialization
// makes it independent of recursion level:
const SecondDerivType x(pi/6,1);
const FirstDerivType sinx = sin(x); // Calculations like normal
const FirstDerivType y = sinx*sinx;
const double raw_y = raw_value(y); // Recursive down-conversion
double deriv = raw_value(y.derivatives());
double deriv2 = raw_value(y.derivatives().derivatives());
Roy H. Stogner Generic Jan 18, 2012 32 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Dual NumbersDefine:
• D[X;n] ≡ {a+∑n
i=1 biεi : a, bi ∈ X}
• Quotient ideal: εiεj = 0 ∀i, j• f(a+
∑biεi) = f(a) +
∑bif
′(a)εi
E.g. with independent variables e.g. x, y, z ∈ R, we set
X ≡ x+ ε1
Y ≡ y + ε2
Z ≡ z + ε3
and get:
f(X,Y, Z) = f(x, y, z) +∂f
∂x(x, y, z)ε1 +
∂f
∂y(x, y, z)ε2 +
∂f
∂z(x, y, z)ε3
Roy H. Stogner Generic Jan 18, 2012 33 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Dual NumbersDefine:
• D[X;n] ≡ {a+∑n
i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j
• f(a+∑biεi) = f(a) +
∑bif
′(a)εi
E.g. with independent variables e.g. x, y, z ∈ R, we set
X ≡ x+ ε1
Y ≡ y + ε2
Z ≡ z + ε3
and get:
f(X,Y, Z) = f(x, y, z) +∂f
∂x(x, y, z)ε1 +
∂f
∂y(x, y, z)ε2 +
∂f
∂z(x, y, z)ε3
Roy H. Stogner Generic Jan 18, 2012 33 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Dual NumbersDefine:
• D[X;n] ≡ {a+∑n
i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j• f(a+
∑biεi) = f(a) +
∑bif
′(a)εi
E.g. with independent variables e.g. x, y, z ∈ R, we set
X ≡ x+ ε1
Y ≡ y + ε2
Z ≡ z + ε3
and get:
f(X,Y, Z) = f(x, y, z) +∂f
∂x(x, y, z)ε1 +
∂f
∂y(x, y, z)ε2 +
∂f
∂z(x, y, z)ε3
Roy H. Stogner Generic Jan 18, 2012 33 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Dual NumbersDefine:
• D[X;n] ≡ {a+∑n
i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j• f(a+
∑biεi) = f(a) +
∑bif
′(a)εi
E.g. with independent variables e.g. x, y, z ∈ R, we set
X ≡ x+ ε1
Y ≡ y + ε2
Z ≡ z + ε3
and get:
f(X,Y, Z) = f(x, y, z) +∂f
∂x(x, y, z)ε1 +
∂f
∂y(x, y, z)ε2 +
∂f
∂z(x, y, z)ε3
Roy H. Stogner Generic Jan 18, 2012 33 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Dual NumbersDefine:
• D[X;n] ≡ {a+∑n
i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j• f(a+
∑biεi) = f(a) +
∑bif
′(a)εi
E.g. with independent variables e.g. x, y, z ∈ R, we set
X ≡ x+ ε1
Y ≡ y + ε2
Z ≡ z + ε3
and get:
f(X,Y, Z) = f(x, y, z) +∂f
∂x(x, y, z)ε1 +
∂f
∂y(x, y, z)ε2 +
∂f
∂z(x, y, z)ε3
Roy H. Stogner Generic Jan 18, 2012 33 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Recursive TemplatesExample
typedef DualNumber<RawType,NumberArray<3,RawType> > FirstDerivType;
typedef DualNumber<FirstDerivType,NumberArray<3,FirstDerivType> >
SecondDerivType;
NumberArray<3, FirstDerivType> xyz;
xyz[0] =
FirstDerivType(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FirstDerivType(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FirstDerivType(z,NumberArrayUnitVector<3,2,RawType>::value());
const NumberArray<3, FirstDerivType> vec = some_function(xyz);
RawType divergence_vec = 0;
for (unsigned int i=0; i != 3; ++i)
divergence_vec += vec[i].derivatives()[i];
Roy H. Stogner Generic Jan 18, 2012 34 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Recursive TemplatesExample
typedef DualNumber<RawType,NumberArray<3,RawType> > FirstDerivType;
typedef DualNumber<FirstDerivType,NumberArray<3,FirstDerivType> >
SecondDerivType;
NumberArray<3, FirstDerivType> xyz;
xyz[0] =
FirstDerivType(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FirstDerivType(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FirstDerivType(z,NumberArrayUnitVector<3,2,RawType>::value());
const NumberArray<3, FirstDerivType> vec = some_function(xyz);
RawType divergence_vec = 0;
for (unsigned int i=0; i != 3; ++i)
divergence_vec += vec[i].derivatives()[i];
Roy H. Stogner Generic Jan 18, 2012 34 / 44
Generic Programming Automatic Differentiation
Gradient Differentiation via Recursive TemplatesExample
typedef DualNumber<RawType,NumberArray<3,RawType> > FirstDerivType;
typedef DualNumber<FirstDerivType,NumberArray<3,FirstDerivType> >
SecondDerivType;
NumberArray<3, FirstDerivType> xyz;
xyz[0] =
FirstDerivType(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FirstDerivType(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FirstDerivType(z,NumberArrayUnitVector<3,2,RawType>::value());
const NumberArray<3, FirstDerivType> vec = some_function(xyz);
RawType divergence_vec = 0;
for (unsigned int i=0; i != 3; ++i)
divergence_vec += vec[i].derivatives()[i];
Roy H. Stogner Generic Jan 18, 2012 34 / 44
Generic Programming Automatic Differentiation
Vector Calculus via Recursive Templates
Playing with vectors becomes easier with various helper functions:
Example
// divergence of both vectors and tensors
template <std::size_t size, typename T>
inline
typename DerivativeType<T>::type
divergence(const NumberArray<size, T>& a)
{
typename DerivativeType<T>::type returnval = 0;
for (unsigned int i=0; i != size; ++i)
returnval += DerivativesOf<T>::derivative(a[i], i);
return returnval;
}
Roy H. Stogner Generic Jan 18, 2012 35 / 44
Generic Programming Automatic Differentiation
MASA PDE Examples
Manufactured Solution
// Arbitrary manufactured solution
U.template get<0>() = u_0 + u_x * sin(a_ux * PI * x / L) +
u_y * cos(a_uy * PI * y / L);
// Why not U[0] and U[1]? To be explained in the future...
U.template get<1>() = v_0 + v_x * cos(a_vx * PI * x / L) +
v_y * sin(a_vy * PI * y / L);
ADScalar RHO = rho_0 + rho_x * sin(a_rhox * PI * x / L) +
rho_y * cos(a_rhoy * PI * y / L);
ADScalar P = p_0 + p_x * cos(a_px * PI * x / L) +
p_y * sin(a_py * PI * y / L);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
Roy H. Stogner Generic Jan 18, 2012 36 / 44
Generic Programming Automatic Differentiation
MASA PDE Examples
Euler
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U)) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U));
Roy H. Stogner Generic Jan 18, 2012 36 / 44
Generic Programming Automatic Differentiation
MASA PDE Examples
Navier-Stokes
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U) - Tau) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U + q - Tau.dot(U)));
Roy H. Stogner Generic Jan 18, 2012 36 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!
Roy H. Stogner Generic Jan 18, 2012 37 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!
Roy H. Stogner Generic Jan 18, 2012 37 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!
Roy H. Stogner Generic Jan 18, 2012 37 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!
Roy H. Stogner Generic Jan 18, 2012 37 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!
Roy H. Stogner Generic Jan 18, 2012 37 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!
I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?
• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)
I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?
• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?
• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?
• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?
• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:
I Sparse vector data types
I Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizations
I Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
Remaining Problems
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• Long-term solution: partial specialization of recursive DualNumber
• Current solution for large n: it’s just going to slow down MASA, so whocares?
I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:
I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations
Roy H. Stogner Generic Jan 18, 2012 38 / 44
Generic Programming Automatic Differentiation
To Be Continued
Roy H. Stogner Generic Jan 18, 2012 39 / 44
Stuff We Won’t Get To This Time Advanced Functional Programming with Templates
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 40 / 44
Stuff We Won’t Get To This Time Functional Data Structures
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 41 / 44
Stuff We Won’t Get To This Time Physics as a Graph Problem
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 42 / 44
Stuff We Won’t Get To This Time Evaluations on the Graph
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 43 / 44
Stuff We Won’t Get To This Time Differentiation on the Graph
Outline
1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation
2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph
Roy H. Stogner Generic Jan 18, 2012 44 / 44