floating-point mixed precision tuning by static analysis

IntroductionPreliminary

Forward & Backward Static AnalysisGroundwork on Constraints

Preliminary ResultsFuture Studies

Floating-Point Mixed Precision Tuningby Static Analysis

FEANICSES 2019

Dorra Ben KhalifaSupervisors : Matthieu Martel & Assalé Adjé

University of Perpignan

21 juin 2019

1 / 34




Mixed Floating-Point Precision Trade-offState of the ArtOutline

Energy Consumption Concern

Top 500 supercomputers energy consumption ' $400 million/yearHow to increase energy efficiency ?

Green Computing : http ://www.green500.orgReduce application energy consumptionSacrifice accuracy for performance⇒ Floating-point precision tuning

2 / 34





What About...

Computer architectures support multiple levels of precision• Higher precision : improves accuracy• Lower precision : reduces energy, running time and bandwidth capacity

Automatically tune floating-point precision is challenging• Without affecting correctness• Improving performance

Precision vs Accuracy !Precision : number of bits representing a value (its format)Accuracy : how close a floating-point computation comes to the real value !

3 / 34





Related Work

TWISTStatic analysis by constraints generationTWIST [3]

CRAFT, Precimonious/HiFPTunerSearch based methodsCRAFT [Lam’13 et al.], Precimonious/HiFPTuner [Rubio’13 et al.] [2]

FPTuner, Rosa/DaisyRigorous error analysis methodsFPTuner [Chiang’17 et al.], Rosa/Daisy [Darulova’14 et al.]

Herbie, SalsaAutomatically discovering unstable floating-point operations andapplying transformationsHerbie [Panchekha’14 et al.], Salsa [DM18]

4 / 34





Overview of our Approach

Forward &Backwardanalysis

Static analysisby constraints generation

Final precision requirement

Tuned optimized program(s)

Output

User specified accuracy

Input

Oversized Precision Program

5 / 34





Outline

1 Preliminary

2 Forward & Backward Static Analysis

3 Groundwork on Constraints

4 Preliminary Results

5 Future Studies

6 / 34




Basic concepts on Floating-Point NumbersUfp and Ulp Functions

Outline

1 Preliminary




5 Future Studies

7 / 34





Basic Concepts on Floating-Point Numbersmantissa mSign s exponent e

precision p

A floating-point number x in base β :

x = s.m.βe−p+1

• s the sign, m the mantissa, e the exponent encoded in the bit string and pis the format precision

IEEE-754 Formats

format bit widthmantissa size(p - 1) exponent size bias

binary16 16 10 5 15binary32 32 23 8 27binary64 64 52 11 1023binary128 128 112 15 16383

8 / 34





Ufp and Ulp Functions

Weight of the most significant bit :

ufp(x) = min{i ∈ N : 2i+1 > x} = blog2(x)c

Weight of the least significant bit :

ulp(x) =

{e− p round to nearest,

e + 1− p otherwise.

Fp : Set of floating point numbers : |v− v̂| ≤ 2e−p+1

∀x ∈ Fp, ulp(x) = ufp(x)− p + 1

Error on x :ε(x) ≤ 2ulp(x)

s e 2e 2 e-p+1

ufp ulp

9 / 34




Forward & backward Analysis for Arithmetic ExpressionsAbstract DomainConcrete Addition in FpAbstract Addition in IpConcrete Multiplication in Fp

Outline

1 Preliminary




5 Future Studies

10 / 34





Forward & backward Analysis for Arithmetic Expressions

+

x

y forward

r

11 / 34






+

x

y forward

r

User requirement

12 / 34






+

x

y forward

r

User requirement

+

x

y Backward

p

Generalizable technique into sets of values !

13 / 34





Abstract Domain

Abstract Values : [a, b]p interval of Fp

e.g : x, y ∈ [1.0, 3.0]16, |v− v̂| ≤ 2ufp(x)−15

Concretization function :

γ([a, b]p) = x ∈ Fp : a ≤ x ≤ b

Partial order :

[a, b]p v [c, d]q ⇔ [a, b] ⊆ [c, d] ∧ q ≤ p

[a, b]p is more precise than [c, d]q with a greater accuracy

14 / 34





Concrete Addition in Fp

+

x

y

p'

q'

r'

ufp(Ɛx+Ɛy)ufp(x+y)

Forward addition :p′ : size of εx, q′ : size of εy−→⊕(xpp′ , yqq′ ) = zrr′ with r = ufp(x + y)− ufp(ε(x) + ε(y))

Backward addition :←−⊕(zrr′ , yqq′ ) = (z− y)pp′ avec p = ufp(z− y)− ufp(ε(z)− ε(y))

15 / 34





Abstract Addition in Ip

−→�([1.0, 3.0]16, [1.0, 3.0]16) = [2.0, 6.0]16

←−�([2.0, 6.0]10, [1.0, 3.0]16) = [1.0, 3.0]9

+

16 bits

17 bits

ufp ulp

=

3#16

1#16

4#17

1 1

1 0 0

0

1 0 0 0 0 0

0 0 0

0 0 0 0 0 0 0 0

000000

0 0 0 0 0 0

0

−→⊕(1.016, 1.016) = 2.016−→⊕(1.016, 3.016) = 4.017

−→⊕(3.016, 1.016) = 4.017−→⊕(3.016, 3.016) = 6.016

16 / 34





Concrete Multiplication in Fp

Forward multiplication :−→⊗(xpp′ , yqq′ ) = zrr′ where r = ufp(x× y)− ufp(ε(x× y))

and ufp(ε(x× y)) = y.ε(x) + x.ε(y) + ε(x).ε(y)

Backward multiplication :

←−⊗(zrr′ , yqq′ ) = (z÷y)rr′ where p = ufp(z÷ y)− ufp(

y.ε(zr)− z.ε(yq)

y.(y + ε(yq))

)

NoteProblem reduced to a system of constraints made of linear relations betweeninteger elements only

17 / 34




StrategySystematic Constraint Generation

Outiline

1 Preliminary




5 Future Studies

18 / 34





Before Constraint Generation....

Preliminary range measure by static analysis (no overflow)The accuracy may↙ in forward analysis Weaken the pre-conditionsThe accuracy may↗ in backward analysis Strengthen thepost-conditions

z = x� y with � ∈ {+,−,×, /}

lower(AccB(z)) =

lower AccB(x) in order to lower AccB(z)lower AccB(y) in order to lower AccB(z)lower both AccB(x) and AccB(y)

19 / 34





Systematic Constraint Generation

Expression : e : := c] p` | id` | e`11 +` e`2

2 | e`11 −` e`2

2 | e`11 ×` e`2

2 | e`11 ÷` e`2

2Boolean : b : := true | false | e`1

1 <` e`22 | e`1

1 >` e`22 | e`1

1 =` e`22

Statement : c ::= c`11 ; c`2

2 | id =` e`1 | while` b`0 do c`11 | if ` b`0 then c`1

1 else c |require_accuracy(x,n)`

l ∈ Lab unique label is attached to each expression and statementΛ : Id → Id × Lab, x =l e`1

We assign to each label l three variables : accB(l), accF(l) and acc(l)

0 ≤ accB(l) ≤ acc(l) ≤ accF(l)

20 / 34





Case of the Forward Addition (1/2)

a = ufp(x) b = ufp(y)

ε(x) ≤ 2a−p+1 ε(y) ≤ 2b−p+1 ε+ < 2a−p+1 + 2b−p+1

Definition :

+

ι = 1

ulp(εx)

ufp(εx)

y+

x

y

ι = 0

ulp(εx)

ufp(εx)

x

ι(ε(x), ε(y)) =

{0 if ulp(ε(x)) > ufp(ε(y))1 otherwise.

Lemma 1 :

ufp(ε+) ≤ max(a− p, b− q) + ι(a− p, b− q)

r+ = ufp(x + y)− max(a− p, b− q)− ι(a− p, b− q)21 / 34





Case of the Forward Addition (2/2)

A = ufp(εx) B = ufp(εy) C = ufp(εz)

How to compute r′ = ufp(ε(z))− ulp(ε(z))?

ufp(Ɛz) ulp(Ɛz)

r'

We have :

U = ufp(εz) and u = ulp(εz)

U = ufp(z)− R

u = min{

ufp(x)− p− p′ + 1ufp(y)− q− q′ + 1.

22 / 34





Case of the Forward Multiplication

ε(x) ≤ 2a−p+1, ε(y) ≤ 2b−p+1

ufp(ε×) ≤ 2a+1.2b−q+1 + 2b+1.2a−p+1 + 2a−p+1.2b−q+1

= 2a+b−q+2 + 2a+b−p+2 + 2a+b−p−q+2

ufp(ε×) ≤ max(a + b− p + 2, a + b− q + 2) + ι(p, q)

≤ max(a + b− p + 1, a + b− q + 1) + ι(p, q)

Thus :

r× = ufp(x× y)− max(a + b− p + 1, a + b− q + 1)− ι(p, q)

Linear constraints !

23 / 34




Outline

1 Preliminary




5 Future Studies

24 / 34




Syntax of IMP

Expression : e : := constant | id | e + e | e - e | e × e | e ÷ e

Boolean : b : := true | false | e < e | e > e | e ≤ e | e ≥ e | e = e

Statement : c : := c ; c | id = e | while b do c | if b then c else c

Work Environment :Java SE Development Kit 8

Eclipse IDE Java Oxygen.2 Release (4.7.2)

ANTLR4 IDE Eclipse Plugin for ANTLR 4 1

1. https ://github.com/antlr25 / 34




Example (1/4) : Parsing Tree

26 / 34




Example (2/4) :Constraints Semantic

ε[cl053]Λ = {accF(l0) = 53}

ε[cl253]Λ = {accF(l2) = 53}

ε[xl4 +l6 yl5 ]Λ = C[xl4 ]Λ ∪ C[yl5 ]Λ ∪ F+(l4, l5, l6) ∪ B+(l4, l5, l6)

C[z :=l7 x + yl6 ]Λ = (C,Λ[z→ zl7 ])

C[require_accuracy(z, 25)l9 ]Λ = {accB(Λ(z)) = 25)}

27 / 34




Example (3/4) : Constraints Generation (Program.z3)

28 / 34




Example (4/4) : Z3 SMT solver solution

x|25| = [1.0,2.0]|25|;y|24| = [3.0,4.0]|24|;z|25| = [1.0,2.0]|25| +|25| [3.0,4.0]|24|;require_accuracy(z,25);

Number_Variables= 49

Number_Constraints = 57

Cost functionSolutions are not unique. We need to add an additional constraint related to acost function φ to the constraints

φ(c) =∑

x∈Id,l∈Lab

acc(xl) +∑

l∈Lab

acc(l)

29 / 34




Policy IterationConclusionReferences

Outline

1 Preliminary




5 Future Studies

30 / 34





Policy Iteration

MotivationZ3 SMT solver = decision tool 6=optimization tool

IdeaUsing Policy iteration to improveaccuracy [1]Generated constraints are of the formmin-max of discrete affine mapsFeeding the policy iteration with the z3solution as an initial policy !

FinalityComparing the policy iteration and Z3solutions (in term of execution time andoptimality)

y=x

P1

P2

P3

P4

P5

x1x3x5

Choice of a new policy

31 / 34





Conclusion

Floating-point computations determination minimal precision

Contribution

Forward & Backward static analysis for numerical accuracyFormulation as first order linear constraints

Extensions : functions, arrays, fixed-point arithmetic, etc.

Minimal precision determination : Policy Iteration

Experimentally tool validation : embedded systems, numericalcomputation, etc.

32 / 34





References

Stephane Gaubert, Eric Goubault, Ankur Taly, and Sarah Zennou.Static analysis by policy iteration on relational domains.In Rocco De Nicola, editor, Programming Languages and Systems, pages237–252, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.32

Stef Graillat, Fabienne Jézéquel, Romain Picot, François Févotte, and BrunoLathuilière.PROMISE : floating-point precision tuning with stochastic arithmetic.In 17th international symposium on Scientific Computing, Computer Arithmeticand Verified Numerics (SCAN 2016), pages 98–99, UPPSALA, Sweden,September 2016.5

Matthieu Martel.Floating-point format inference in mixed-precision.In NASA Formal Methods - 9th International Symposium, NFM 2017, MoffettField, CA, USA, May 16-18, 2017, Proceedings.5 33 / 34





THANK YOU FOR LISTENING

Q & A!

34 / 34

floating-point mixed precision tuning by static analysis

Documents