a personal and historical perspective on some uses of least ......a personal and historical...

A personal andhistorical

perspective onsome uses of least

squares

Antoine deFalguerolles

Stephen Stigler,1999

Box and Draper,1987

Data structure

RegressionLaplace, 1796

Tobias Mayer, 1748

Legendre, 1805

Georges Müntz, 1834

Augustin-Louis Cauchy,1836

Ibáñez and Saavedra, 1859

Pareto, 1897

SVDLa Armada Invencible, 1558

Principal coordinatesanalyses

Horse kicks, 1898

Correspondence Analysis

Perspectives?IW-SVD

A ghost model for MCA

The end

A personal and historical perspective onsome uses of least squares

Antoine de Falguerolles

Université de Toulouse III - Paul Sabatier (France)Dr hab. (retired)

antoine @ falguerolles.net



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Summary

As observed by Stephen Stigler in one of is influential book“The method of least squares is the automobile of modernstatistical analysis [...] this method and its numerousvariations [...] carry the bulk of statistical analyse”. In mypresentation, I will consider a series of situations (eitherregression or singular value decomposition) in which leastsquares or some of its variations have been actually used orwould be used nowadays. In all cases, some light modeling isused as a safeguard against separated evolutions ofstatistical “Schools” (e.g. use of the R package gnm forcorrespondence analysis).



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Stephen Stigler’s full citation

Chapter 17Gauss and the invention of Least Squaresp. 320

The method of least squares is the automobileof modern statistical analysis: [despite itslimitations, occasional acccidents, and incidentalpollution,] this method and its numerousvariations, extensions and related conveyances]carry the bulk of statistical analyses[, and areknown and valued by nearly all].

Stephen Stigler (1999): Statistics on the Table, Cambridge:Harvard university Press.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

George Box and Norman Draper, 1987

Essentially, all models are wrong, but some areuseful. However, the approximate nature of themodel must always be borne in mind.

George Box and Norman Draper (1987): Empirical modelbuilding and response surfaces, New York: John Wiley andsons (p. 424 ?).



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Data structureYan r× c response matrix

Xrinformation on the rowsand names

X ′cinformation on thecolumns and names

Missing dataWeightsImpact of the coding

Y Xr

X ′c



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Hint to Pierre Legendre(1995)’s fourth-cornerproblem where rowsare species and columnsites.

Y Xr

X ′c ?



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Table of contentsStephen Stigler, 1999Box and Draper, 1987Data structureRegression

Laplace, 1796Tobias Mayer, 1748Legendre, 1805Georges Müntz, 1834Augustin-Louis Cauchy, 1836Ibáñez and Saavedra, 1859Pareto, 1897

SVDLa Armada Invencible, 1558Principal coordinates analysesHorse kicks, 1898Correspondence Analysis

Perspectives?IW-SVDA ghost model for MCA

The end



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Least squares in what we call a regression context appearedbetween 1795 (Gauss’s claims but no published evidence)and 1805 (Legendre’s publication).Still

I Before? (18th Century)I After? (19th Century)

I generalised least squaresI qualitative explanatory variableI generalised linear model



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Laplace (an IV)(23 september 1795 - 21 septembre 1796)

Mathématiques

a lesson taught on the 21 Floréal de l’An IV

(20 May 1796), pp. 67-68



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

An enigmatic assertion on the computing of a central value.Noticeable ideas:

I weight (poids): wiI influence of the error (influence de l’erreur) computed

by differential calculus: hiI wi = 1hiI gravity center (un centre de gravité):

The context is astronomical (and my language surelyanachronistic)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Let yi (i ∈ {1, . . . , n}) be observed values and xi (xi > 0)the corresponding computed values such that yi ≈ bxi, bbeing an unknown correction coefficient.How do we estimate b? A “middle”of the results of severalobservations (fixer un milieu entre les résultats de plusieursobservations)?

yixi, i ∈ {1, . . . , n}

Take b as the centre of gravity of the weighted yixi

b =∑i=1,...,nwi(x1, . . . , xn)

yixi∑

i=1,...,nwi(x1, . . . , xn)

wi(x1, . . . , xn) for the yixi ? The weights could also dependon the yis. But not really considered here.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Remember, without much loss of generality, xi > 0Exemples

I wi(x1, . . . , xn) = 1,

b = 1n

∑i=1,...,n

yixi

I wi(x1, . . . , xn) = xi,

b =∑i=1,...,n xi

yixi∑

i=1,...,n xi

I wi(x1, . . . , xn) = x2i ,

b =∑i=1,...,n x

2iyixi∑

i=1,...,n x2i



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

I wi(x1, . . . , xn) = 1,

b = 1n

∑i=1,...,n

yixi

I wi(x1, . . . , xn) = xi,

b =∑i=1,...,n yi∑i=1,...,n xi

I wi(x1, . . . , xn) = x2i ,

b =∑i=1,...,n xiyi∑i=1,...,n x

2i



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Caveat: An interpretation extracted from further publicationsby Laplace (and discussions with patient colleagues)!Influence of the error in independent observations onthe unknown coefficient?

I Model 1: e = b− yx , h =dbde = 1 and w = 1

I Model 2: ex = b−yx , h =

dbde =

1x and w = x

Model 2 and Laplace’s suggestion lead to the so-calledestimator of Roger Cotes (1682 - 1716):

a =∑i=1,...,n xi

yixi∑

i=1,...,n xi=∑i=1,...,n yi∑i=1,...,n xi

a linear estimator but not the yet unknown least-squaressolution for the regression model y ≈ bx.Note: if the yi are observed values of independent binomial B(xi, π), then the rule of Roger Cotes

gives the maximum likelihood estimator of π.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Exercise: Let Zi ∼ L(β, σ2

x2i) be n independent r.v.. Compare

the two following unbiased estimators of β:

1n

∑1,...,n

Zi and1∑

1,...,n |xi|∑

1,...,n|xi|Zi

Hint: xi > 0, mA1 ≥ mH1 > 0, 1(mA1 )2 ≤1

(mH1 )2≤ 1

mH2where

mAk is the arithmetic mean of kth powers of values and mHkthe harmonic mean of the kth powers.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

And multiple regression?

the situation had been already considered!



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Tobias Mayer 1748

The ordinary multiple regression model (in the anachronisticvector and matrix form):

E[Y ] = Xβ and Var(Y ) = σ2I

U is any conformable matrix (U ′ and β, U and X have samenumber of rows).An analogue of the normal equations is the system:

U ′Xβ = U ′Y

If U ′X is regular, β̂M = (U ′X)−1U ′Y isI a linear and unbiased estimator of βI Var(β̂M ) = σ2(U ′X)−1U ′U(X ′U)−1



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Several ways of constructing the analogues of matrix U andsystem (U ′X)β̂M = U ′Y

I Mayer: if p regression coefficients, then split the datainto p clusters; aggregate the data in each clusters toobtain a system of linear equations of p equations and punknown.

I Ideas from Laplace’s lesson: E[Y ] = Xβ = [xji ]β; forall column j of X, multiply every line i ofY − [X]β = 0 by sign(xji )|x

ji | and add all revised lines

into one linear equation.I Laplace: overlapping clusters; linear combinations with

coefficients in {−1, 0, 1}.Potential computational problem: solving the linearsystem p× p by hand

(U ′X)β̂M = U ′Y β̂M = σ2(U ′X)−1U ′Y

Triangularisation.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Legendre, 1805

The first page of Legendre’s appendix on least squares. Theunknown coefficients are x, y, z . . . , the variables are a, b, c,f . . . Note that a (or rather −a) is the response variable, andthat there is no formal intercept (Preliminary centering?).E, the error, is zero or a very small quantity.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Measuring Paris meridian arc

A famous data set in metrology!

Place Arc betweenLatitude L’-L L’+L

in modules

Dunkerque 51˚ 2’10”50DP 62472.59 2˚11’20”75 99˚53’ 0”

Panthéon 48˚50’49”75PE 76145.74 2˚40’ 7”25 95˚ 1’32”

Evaux 46˚10’42”50EC 84424.55 2˚57’48”10 89˚23’37”

Carcassonne 43˚12’54”40CM 52749.48 1˚51’ 9”60 84˚34’39”

Montjouy 41˚21’44”80

See Stephen M. Stigler, “Gauss and the invention of leastsquares”, The annals of statistics, 1981, 9(3), 465–474.

I i (i = 0, . . . , 4) the places of observation.I Li their latitudes.I Si the length of the 4 arcs i (i = 1, . . . , 4) between the places of latitudes Li−1 et Li.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

The equations to match:two potential models under consideration, one preferred byLegendre (number 2, in red).

1. Response variable: yi = Si. Linear predictor:µi(β) = β1(Li − Li−1) + β2K sin(Li − Li−1) cos(Li + Li−1)

where K is a known coefficient, and where β1 et β2 denote the unknown regression coefficients.

2. Response variable : yi = Li − Li−1. Linear predictor:

µi(β) =SiK ′

+β1SiK ′

+β2K ′′ sin(Li−Li−1) cos(Li+Li−1)

where K ′ et K ′′ are known coefficients, where SiK ′

is an“offset”, and where β1 et β2 again denote the unknownregression coefficients.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Matching (Loss) criterion for the equations yi − µi(β) = 0?Ordinary least squares:

arg minβ||y − µ(β)||2

Not quite, since the errors are likely to be correlatedalong the meridian. Legendre’s proposal:

ui = ei − ei−1

Generalised least squares:arg min

β||y − µ(β)||2W

Solved by OLS on augmented data! EM ?

Obvious extensions: MA(1) and AR(1) disturbances,ui = ei + θei−1 (θ ∈ [−1, 1]), and ui = θui−1 + ei(θ ∈]− 1, 1[). (For the latter remember Durbin-Watson!).RegArima?



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Müntz’s data

Group Unit Cost (y) Distance (x)

1 1 18.8 5.501 2 0.8 0.551 3 2.2 1.001 4 12.3 4.001 5 9.9 3.501 6 9.0 3.001 7 10.2 4.001 8 8.4 3.001 9 5.0 2.701 10 2.2 1.201 11 2.7 1.501 12 2.1 1.001 13 1.6 0.801 14 1.5 1.001 15 13.3 4.10

2 1 7.6 3.402 2 6.7 3.352 3 8.9 4.50

Georges Müntz (1807-1887), a road engineer, ontransportation costs of stones.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Covariance analysis

0 5 10 15 20

12

34

5

distance

tran

spor

tatio

n co

st



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Covariance analysis

Y=c(Y1,Y2) ; X=c(X1,X2)G=factor(rep(c(1,2),c(length(Y1),length(Y2))))

> lm(Y˜G:X)$coefficients(Intercept) G1:X G2:X

0.5966443 0.2785730 0.4096165

> lm(Y1˜X1)$coefficients(Intercept) X1

0.6040630 0.2778905

> mean((Y2-lm(Y1˜X1)$coefficients[1])/X2)[1] 0.4051578

Optimality? Gauss-Markov theorem gives the answer.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Augustin-Louis Cauchy, 1836

Augustin-Louis Cauchy (1789-1857) is one of the greatestFrench mathematician of the XIXth century. He wrote about800 papers in an astonishing variety of areas. Hisuncompromising and reactionary (légitimiste) personalityhindered his academic career. He published his regressionmethod while in exile in Prague.

Cauchy’s motivation: alleviate the computional burdenof least squares (cross products and system of linearequations)

The methods of Cauchy and Legendre were oftensimultaneously considered in French applications until theearly 20th century.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Estimation : Cauchy? Cauchy vs Legendre?

I the yi, i ∈ {1, . . . , n}, are observed values ofindependent random variables Yi, Yi ∼ L(µi, σ2)

I a linear model for µisimple regression:

µi = β0 + β1xi

multiple regression:

µi = β0 + β1x1i + . . .+ βKxKi

I a stepwise inclination: introduction of x1, x2, . . . in turn(with some idea on the order of introduction)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Simple regression: µi = β0 + β1xi

Legendre{

β̂1 =

∑ni=1

(xi−x)(Yi−Y )∑ni=1

(xi−x)2

β̂0 = Y − β̂1x

Cauchyβ̂1 =

∑ni=1 sign(xi−x)(Yi−Y )∑n

i=1 |xi−x|

β̂0 = Y − β̂1x

where Y = 1n∑ni=1 Yi, x = 1n

∑ni=1 xi,

and sign(xi − x) = 1(xi − x ≥ 0)− 1(xi − x ≤ 0).

Property: Both methods provide linear unbiased estimators.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Cauchy as weighted Legendre

β̂1 =

∑ni=1 sign(xi−x)(Yi−Y )∑n

i=1 |xi−x|=∑n

i=1 wi(xi−x)(Yi−Y )∑ni=1 wi(xi−x)

2

β̂0 = Y − β̂1x

Cauchy and Mayer’s method of meansβ̂1 =

1∑ni=1 1(xi>x)

∑ni=1 Yi1(xi>x)−

1∑ni=1 1(xix)−

1∑ni=1 1(xi



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Multiple regression: µi = β0 + β1x1i + β2x2iCommon procedure with specific estimation (Legendre, orCauchy, or whatever).

I Step 0. All variables are mean centered: y(0)i = yi − y,x

1,(0)i = x1i − x1, x

2,(0)i = x2i − x2.

I Step 1. Simple regressions of y(0) on x1,(0) with slopeby1, and of x2,(0) on x1,(0) with slope b21. Computeresiduals y(1)i = y

(0)i − b

y1x

1,(0)i and

x2,(1)i = x

2,(0)i − b21x

1,(0)i

I Step 2. Simple regression of y(1) on x2,(1) with slopeby2. Compute residuals y

(2)i = y

(1)i − b

y2x

2,(1)i

Finally: b2 = by2, b1 = by1 − b

y2b

21, b0 = y − b1x1 − b2x2

(triangularisation).If LS in each step, final result independent from the entering order of the explanatory variables.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

One slope, several intercepts

I Carlos Ibáñez eIbáñez de Ibero(Barcelona, 1825 -Niza, 1891)

I Frutos SaavedraMeneses (1823 -1868)

I Aimé Laussedat(1819-1907) (1860)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

One slope, several intercepts

µi,g(i) = βg(i),0 + β1xi,g(i)This problem arose in the calibration of some parts of anapparatus to be used in geographical triangulation.The number of levels for the group factor g is large and thedesign is balanced.Least squares are used.The block pattern of the design matrix and the normalequations is recognised and closed form estimators areobtained.Residuals are carefully checked.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Pareto, 1897

Pareto (1848-1923) was born in Paris from an exiled Italianfather and a French mother. A graduate from thePolitecnico di Torino, he was appointed a lecturer ineconomics at the University of Lausanne in Switzerland in1893. He died in Switzerland.

In a 1897 paper in the Journal de la Société de statistique deParis

I Artificial dataI Example 1: annual variation of population size over time in England and Wales (regression

issues)I Example 2: annual covariation over time of the number of weddings and economic prosperity in

England; prosperity is measured by two covariates: coal extraction and exports (differencing oftime series issues)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

population growth

Year Population(i) (yi)

1 88932 101643 120004 138975 159141 179287 200668 227129 25974

10 29003

Notation:i = 1, . . . , 10yi population size in year xi = i

Modeling:Yi ∼ L(µi, σ2)

I Arithmetic progressionµi = β0 + β1xi(identity link)

I Geometric progressionlog(µi) = β0 + β1xi(log link)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Minimization of ∑ni=1(yi − exp{β0 + β1xi})2)Gradiant

∇S(β0, β1) = −[ ∑n

i=1(yi − µi)µi∑ni=1(yi − µi)µixi

]

HessianHS(β0, β1) =

−

[ ∑ni=1

((yi − µi)µi − µ2i

) ∑ni=1

((yi − µi)µixi − µ2ixi

)∑ni=1

((yi − µi)µixi − µ2ixi

) ∑ni=1

((yi − µi)µix2i − µ

2ix

2i

) ]

At a stationnary point (β?0 , β?1)′, ∇S(β?0 , β?1)′ = (0, 0)′, then

HS(β?0 , β?1) =n∑i=1

µ?i2[

1 xixi x

2i

]−

n∑i=1

µ?ix2i

[0 00 yi − µ?i

]



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Minimization of ∑ni=1(yi − exp{β0 + β1xi})2Pareto searches a stationnary point using a Newton-Raphsonalgorithm in which a modified Hessian is used

[β

(k+1)0β

(k+1)1

]=[β

(k)0β

(k)1

]− H̃S(β(k)0 , β

(k)1 )−1∇S(β

(k)0 , β

(k)1 )

whereH̃S(β?0 , β?1) =

n∑i=1

µ?i2([

1 xixi x

2i

])

Has Pareto realized that E[(Yi − µi)µix2i ] = 0 and that hecan use the so called Fisher’s scoring method (Ronald Fisher,1890-1962) ! Note that Pareto’s simplified Hessian is, underGaussian assumptions, Fisher’s information matrix for β0 andβ1.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Weighted least-squares for the log tranformedresponse

Weighting the log-transformed observations might provide afair approximation. Pareto suggests to weight each relationlog(yi) ≈ β0 + β1xi by yi, or weights wi = y2i in the lossfunction:

∑ni=1 y

2i (log(yi)− (β0 + β1xi))2.

How it does it relate to IWLS?I Some GLM notation: Yi ∼ L(µi, σ2) with link functiong, g(µi) = ηi = β0 + β1xi. Here g = log

I Associated loss function:∑ni=1(yi − exp{β0 + β1xi})2

I Loss function at iteration k:∑ni=1 (exp{η

(k−1)i })2(z

(k)i − (β0 + β1xi))2

where z(k)i =yi−exp{η(k−1)i }

exp{η(k−1)i }+ η(k−1)i



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Weighted least-squares for the log tranformedresponse

I Initial values: z(0)i = η(0)i = log(yi).

I Loss function at iteration 1:∑ni=1 (yi)2(log(yi)− (β0 + β1xi))2



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Eigen vs Singular

SVD came into the statistical landscape in the 1980s andreplaced EVD in many uses in statistics. Both extends tocomplex matrices.Y is a #r ×#c real matrix of rank K. Consider the optimalapproximation of Y by a matrix of rank k (k ≤ K):

arg minM∈Rk ||Y −M ||2

SVD provides a solution:∑k`=1 d`u

`{v`}′

I SVD is well documented and implementedI useful for factorisationI useful for visualisation

I distances between rows of Y , inner product for Y ′YI biplot



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

La Armada Invencible, 1558V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

A1 12 7737 3330 1293 4623 347 18450 789 186 150A2 14 6567 1937 863 2800 238 11900 477 140 87A3 16 8714 2458 1719 4171 384 23040 710 290 309A4 11 8762 2325 780 3105 240 10200 415 63 119A5 14 6991 1992 616 2608 247 12150 518 139 109A6 10 7705 2780 767 3523 280 14000 584 177 141A7 23 10271 3121 608 3729 384 19200 258 142 215A8 22 1221 479 574 1093 91 4550 66 20 13A9 4 0 873 468 1341 200 10000 498 61 88A10 4 0 0 362 362 20 1200 60 20 20

I Rows (10): fleets forming the Armada: Armada de Galeones de Portugal (A1), Armada deViscaya (A2), Galeones de la Armada de Castilla (A3), . . . , Galeras (A10)

I Columns (10 items): Numero de navios (V1), Toneladas (V2), Gente de guerra (V3), Gente demar (V4), Numero de todos (V5=V3+V4), Pieças di artilleria (V6), Peloteria (V7), Poluora(V8), Plomo qui tales (V9), Cuerda qui tales (V10)

I Total columns sometimes differ from the published total (typographic errors more than grosserrors).

As many columns as rows. Coincidence?



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

SVD

mat.Y=as.matrix(read.csv(’A.csv’, header = TRUE, sep = ","))

vec.m=apply(mat.Y,2,mean)mat.V=cov.wt(mat.Y)$covmat.R=cor(mat.Y)mat.Ys=sweep(mat.Y,2,sqrt(diag(mat.V)),’/’)mat.Yc=sweep(mat.Y,2,vec.m,’-’)mat.Z=sweep(mat.Yc,2,sqrt(diag(mat.V)),’/’)> round(sum(mat.Z*mat.Z),3)[1] 90

> round(sum(mat.Z*mat.Z)-cumsum(svd(mat.Z)$dˆ2),3)[1] 22.185 11.089 5.590 2.912 1.264[6] 0.431 0.063 0.003 0.000 0.000

Two principal dimensions? The rounded third squared sv is5.590.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

gnm

library(gnm);Ys=as.vector(mat.Ys)F.R=factor(row(mat.Y), labels=rownames(mat.Y))F.C=factor(col(mat.Y), labels=colnames(mat.Y))

> deviance(gnm(Ys˜F.C))[1] 90

> deviance(gnm(Ys˜F.C+instances(Mult(F.R,F.C),3)))InitialisingRunning start-up iterations..Running main iterations...........................Done[1] 5.589685



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

SVD

I Data matrix Y (#r ×#c) with rank K, column meanvector y, covariance matrix V and correlation marix R

I Preprocessing Y : Ys column standardized; Yc columncentered; Z = [zcr] = [zr ′] = [zc] column centered andstandardised.

I SVD of Z: Z = Udiag(d)V ′, where: U ′1#r = 0, andU ′U = V ′V = Ik

I LS approximation of Z of order k (k ≤ K):

k∑`=1

d`u`{v`}′



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

gnm

Reconstitution formula for Ys = [ycr]:

ycr = yc +k∑`=1

d`u`rv`c

Sum of a main column effects and a multiplicative (bilinear)interaction.

Note: other identification constraints than the ones used inSVD are possible



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Death by horsekick in Prussian cavalry, 1898

G I II III IV V VI VII VIII IX X XI XIV XV1875 0 0 0 0 0 0 0 1 1 0 0 0 1 01876 2 0 0 0 1 0 0 0 0 0 0 0 1 11877 2 0 0 0 0 0 1 1 0 0 1 0 2 01878 1 2 2 1 1 0 0 0 0 0 1 0 1 01879 0 0 0 1 1 2 2 0 1 0 0 2 1 01880 0 3 2 1 1 1 0 0 0 2 1 4 3 01881 1 0 0 2 1 0 0 1 0 1 0 0 0 01882 1 2 0 0 0 0 1 0 1 1 2 1 4 11883 0 0 1 2 0 1 2 1 0 1 0 3 0 01884 3 0 1 0 0 0 0 1 0 0 2 0 1 11885 0 0 0 0 0 0 1 0 0 2 0 1 0 11886 2 1 0 0 1 1 1 0 0 1 0 1 3 01887 1 1 2 1 0 0 3 2 1 1 0 1 2 01888 0 1 1 0 0 1 1 0 0 0 0 1 1 01889 0 0 1 1 0 1 1 0 0 1 2 2 0 21890 1 2 0 2 0 1 1 2 0 2 1 1 2 21891 0 0 0 1 1 1 0 1 1 0 3 3 1 01892 1 3 2 0 1 1 3 0 1 1 0 1 1 01893 0 1 0 0 0 1 0 2 0 0 1 3 0 01894 1 0 0 0 0 0 0 0 1 0 1 1 0 0



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

CA’s formulas, miracles and confusion

c...

r . . . yRCrc . . . yRr

...yCc y

∅

c...

r . . . eRCrc . . ....

I A two-way table of counts Y RC = [yRCrc ], marginalvectors Y R = [yRc ] and Y C = [yCc ], and total y∅

I table of expected values under additivity of effects(independence): 1

y∅Y RY C

′ = [yRr y

Cc

y∅]

I associated two-way table E of Pearson’s residuals :

E = [eRCrc ] = [yRCrc −

yRr yCc

y∅√yRr y

Cc

y∅

]

Light taste of Poisson or multinomial



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

CA formulas, miracles and confusion

I SVD of E: E = Udiag(d)V ′

I Row and column metrics WR = 1y∅ diag(YR) and

WC = 1y∅ diag(YC)

I Reconstruction formula: Y = [yRCrc ] =1y∅Y RY C

′(1 + 1√(y∅){W−

12

R U}diag(d){W− 12C )V }′

Light taste of Gaussian (LS)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

CA as a gnm

I yRCrc observations of independent r.v. Y RCrcI E[Y RCrc ] = µRCrcI Var(Y RCrc ) = 1y∅ y

Rr y

Cc

I data driven mean (and link) function:µ(ηRCrc ) = 1y∅ y

Rr Y

Cc (1 + ηRCrc )

I rank([ηRCrc ]) = k (k small)I some processing of [ηRCrc ] for visualisation

Note:“Data driven” rather than “canonical” link function.Therefore more troubles in handling missing data!

I am indebted to David Firth and Heather Turner for havinghelped me to implement CA as a gaussian model in gnm



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

Perspectives?

I Flexibility of gnm for implementing standard analysesand many more (possibility to introduce otherexplanatory variables the row effects and columneffects). Modeling with a menagery of interesting formsof interactions

I IW-SVD (Are there simple solutions to K. RubenGabriel and Shmuel Zamir (1979) element-wiseweighted low rank approximation?)

I Multiway table:Why MCA? A ghost model forMCA



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

IW-SVD: Notation and problem

I Triple (Y,WR,WC) whereI Y = [yRCrc ] is #r ×#cI WR metric for the rows: WR positive definite;1

′#rWR1#r = 1

I WC metric: for the column: WC positive definite;1

′#cWC1#c = 1

I yRCrc observed values of independent rv Y RCrcI E[Y RCrc ] = µRCrc = m(ηRCrc )I Var(Y RCrc ) = φV(µRCrc )I [ηRCrc ] is a rank k + 1 matrix:ηRCrc = β∅ + βR0,r + βC0,c +

∑k`=1 β

R`,rβ

C`,c

Minimisation problem:

#r∑r=1

#c∑c=1

(yRCrc −m(ηRCrc ))2

V(µRCrc )



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

IW-SVD iteration

I At step i, matrix [n(i)RCrc ] is the current estimate of[ηRCrc ]

I Working response matrix:Z = [zRCrc ] = [

yRCrc −m(n(i)RCrc )

m′(n(i)RCrc )+ n(i)RCrc ]

I Matrix of working weights [{m′(n(i)RCrc )}2

V (m(n(i)RCrc ))],

W = [√{m′(n(i)RCrc )}2

V (m(n(i)RCrc ))], and W̃ = [

√V (m(n(i)RCrc )){m′(n(i)RCrc )}2

]

Minimisation problem:

#r∑r=1

#c∑c=1

{m′(n(i)RCrc )}2

V (m(n(i)RCrc ))(zRCrc −(β∅+βR0,r+βC0,c+

k∑`=1

βR`,rβC`,c))2

This is exactly the problem raised by Gabriel and Zamir(1979). It was then solved by alternating regressions.



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

IW-SVD troubles

The Schur product by W̃ of a rank approximation of W ◦ Zdoes not preserve the rank.

Exceptions are sometimes possible, e.g. the miracle in CA:V ar(Y RCrc ) = m′(ηRCrc ) = 1y∅

yRry∅

yCcy∅

since it factorizes.

The gnm package offers a practical solution to Gabriel andZamir problem.

Is there an algorithm based on SVD? Working on augmenteddata?



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end


Is MCA, CA of a BURT matrix, a trick? Is there a modelbehind?

I if Y ABCabc ∼ L(µABCabc , y∅yAay∅

yBby∅

yCcy∅

) where:

I µABCabc = y∅yAay∅

yBby∅

yCcy∅

(1 + ηABCabc )I ηABCabc has model:

1+A+B+C+[A×B]k(A,B)+[A×C]k(A,C)+[B×C]k(B,C)

I then Y ABab ∼ L(µABab , y∅yAay∅

yBby∅

) where:

I µABab = y∅yAay∅

yBby∅

(1 + ηABab )I ηABab has model:

1 +A+B + [A×B]k(A,B)



squares



Box and Draper,1987

Data structure


Tobias Mayer, 1748

Legendre, 1805




Pareto, 1897



Horse kicks, 1898


Perspectives?IW-SVD


The end

I have reached the end of my tale.

Thank you for your attention

Stephen Stigler, 1999Box and Draper, 1987Data structureRegressionLaplace, 1796Tobias Mayer, 1748Legendre, 1805Georges Müntz, 1834Augustin-Louis Cauchy, 1836Ibáñez and Saavedra, 1859Pareto, 1897

SVDLa Armada Invencible, 1558Principal coordinates analysesHorse kicks, 1898Correspondence Analysis

Perspectives?IW-SVDA ghost model for MCA

The end

a personal and historical perspective on some uses of least ......a personal and historical...

Documents