This article was downloaded by: [Universitaetsbibliothek Dortmund]On: 15 July 2013, At: 09:07Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street,London W1T 3JH, UK

Statistics: A Journal of Theoretical and Applied StatisticsPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/gsta20

The Hadamard Product and Some of its Applications in StatisticsH. Neudecker a , S. Liu a & W. Polasek aa University of Amsterdam and University of Basel,Published online: 27 Jun 2007.

To cite this article: H. Neudecker , S. Liu & W. Polasek (1995) The Hadamard Product and Some of its Applications in Statistics, Statistics: A Journal ofTheoretical and Applied Statistics, 26:4, 365-373, DOI: 10.1080/02331889508802503

To link to this article: http://dx.doi.org/10.1080/02331889508802503

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications onour platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to theaccuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are theopinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should notbe relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arisingdirectly or indirectly in connection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution,reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions ofaccess and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Statistics 26 ( 1995) 365-373 d: 1995 OPA (Overseas Publishers Association) Reprints available directly from the publisher Amsterdam B.V. Published under license by Photocopying permitted by license only Gordon and Breach Science Publishers SA

Printed in Malaysia

THE ZIADAMARD PRODLTCT AND SOME OF ITS APPLICATIONS IN STATISTICS

H. NEUDECKER, S. LIU and W. POLASEK

Unicersiry of Amsterdam and University of Basel

f Receiued 3 September 1993: in,final,form 29 August 1994 )

Summary. Algehraic properties of the Hadamard product are used to establish some statistical properties.

AMS 1991 subject classifications: 15A69; 62H99.

Key words: Kronecker and Hadamard products. selection matrlx. random vector, expectation, variance.

1. INTRODUCTION

Some results on Kronecker and Hadamard products are collected in Rao and Kleffe (1988) and Magnus and Neudecker (1991). Early uses of the Hadamard product in statistics can be traced back to e.g. Styan (1973) and Neudecker (1975, 1981). Later on the Hadamard product was applied to the estimation of heteroscedastic linear re- gression models by Amemiya (1985). Some new applications on asymptotic dis- tributions and image factor analysis were given recently by Kollo and Neudecker (1993) and Neudecker (l993), respectively. Algebraic properties and applications were extensively reviewed by Horn (1990).

Some statistical properties of the Kronecker product of random vectors were reported in Magnus and Neudecker (1979) and Neudecker and Wansbeek (1983). It is thought to be relevant and useful to study statistical properties of Hadamard products of vectors. In this paper we shall examine this interesting field of research. In Section 2 new results and new proofs for known results are presented. The expectations, variance matrices, density functions of Hadamard products of random vectors, and the moment-generating function under normality of the Hadamard product x . x are given afterwards.

2. SOME ALGEBRAIC DEFINITIONS. PROPERTIES AND RELATIONSHIPS

Let "vec" indicate the vectorization operator, "0" and "." indicate the Kronecker and Hadamard products respectively. We know that A 0 B = {ui j B ) and A . B = {a i j hi j) , where A = { a i j ) and B = {b i j ) . The two products are intimately connected. Three well-known Kronecker properties are (See Rao and Kleffe (1988) or Magnus and

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

366 H NEUDECKER c,t i d

Neudecker (199 1)):

(vec A)' vec B = tr ,4' B, ( 1

vec uh' = h @ u. (3)

Let furthers: = (1,. . . , 1)' be the p-dimensional summation column vector, A,: = I,- A be the diagonal matrix with diagonal elements aii of matrix A, A(h) be the diagonal matrix such that A(h)s = h. Let ei be the ith unit column vector of dimension p, E,,:= eiei, i = 1 , . . . , p. Let K,, (or K) indicate the p2 x p2 commutation matrix which plays an important role in the context involving vecs and the Kronecker product, i.e. Kpp = Cr,= (E,,,@ Elj). Let J, indicate the pZ x p selection matrix that links the Hadamard and Kronecker products.

It is interesting to mention two seemingly different but identical definitions of J,, viz

P

J,: = C (c, 8 Eii) = ( E l l . . . E,,,)'; i = 1

J,:= C ( el @ vec E,,) = (vec E l l . . . vec E,,). i = 1

For (4) see Rao and Kleffe ( 1 988), for (5) see Kollo and Neudecker ( I 993) or Neudecker (1983,1993). I1 is not difficult to prove that the two definitions are identical. We notice by using h' @ a = u @ h' = ah' and (3) that

hence

P P

(vec E l l . . . vec E,,) = (e; 8 vec E,,) = (ei @ E,,) = (E, , . . . E,,)'. i = 1 i = 1

We now give some properties.

PI. (i) u = JivecA; (ii) vec A, = J,u;

(iii) Ji Jp = I , ; (iv) K,, J , = J,; (v) J, JL = K,;

(vi) . I , Ji vec A, = vec A,,

where a: = Ads, the p x p matrix K, is the diugonul matrix derivedfrom the commutation matrix K .

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

THF. HADAMARD PRODlJCT 367

(See also Rao and Kleffe ( I 988) for ( i ) and (ii). See also Neudecker (1983) for (i) and (iii). See Magnus (1988) for jiii) and (vi). See Kollo and Neudecker (1993) for (iv) and (v).)

(vecE,,)'vecA = trE,! A = r ( Ar = u,;, by using (1); P P

/,a = x aiivecEii = vec x uii Eii = vec A,, using ( 5 ) and thc definilivn of A,; i = l i - 1

l;l.Ipr = .IAvecA(x) = x, for any vector x, by using (i) and (ii); alternatively, 1,; Jp = J,:(1 O I ) J p = 1.1 = I. using P2 (to follow); Using (5 ) ; -

Using (4); .Ip .I; vec A, = .Ira = vec A,.

This establishes the result. . P2. J b ( A @ R ) .Iy = A . l3,ji)r uny p x q mufricw A and B. (See also Browne (1974), Pukelsheim (1977) and Kollo and Aeudecker (1993) for p = q.)

P Y

= 2 a , j bij Eij = A.B, using (4). . ( = I j = 1

P3. J, JA ( r @ Al J, = cr@ Al J,.for any diauonal matrices r a n d A of' order p. (See also Kollo and Neudecker (1993) and Magnus (1988).)

P P P

Proof J, J b ( r @ A) J, = ( E , , @ Ei,)(rC3A) x (ejOEij) = 1 (EiiTe,@ EiiAEii) r = l j= 1 i , j= 1

P

= x yilbi(ei @I E, , ) = (I-@ A) J,, using (4). . r = l

P4. rl AP2.Al BA, = PI Al(A.B)r,A2, ,for any p x q matrices A and B, and compat- ible diugonal mutricrs TI , T2, A, and A,. (See also Styan (1973) for PI = A, = I and Theorem 3.8 (c) in Magnus and Neudecker (1991) for r, = A2 and r, = A l =I.)

Proyf Consider p x q matrices A and B. Then T1AT2.A,BA2 = J b ( r l A T , @ A l B A 2 ) J q = J ~ ( r , @ A l ) ( A @ B ) ( ~ 2 @ A 2 ) J , = J , ( r l @ A l ) Jp J;(A@ B) J, J;(T2 @A2) J, = ( r l .Al ) (A.B)( r2 .A2) = r, Al(A.B)r2A,, by virtue of P2 and P3. . P5. (A@ B).(C@ D) = (A.C)@(B.D). (See also Rao and Kleffe ( l988).)

Proof (A@ B) . (C@ D) = (a , ,B)~{c, ,D) = (a,,c,,B.D) = (A.C)@(B.D). .

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

368 H. NEUDECKER ct cil

P6. (i) (A @ B), = A, @ B,,for arhitrury square matrices A and B. (ii) ( A @ B),s = LI @ h, bvhere u: = Ads trnd h: = B,s ,f& urhitrurj, squure murriccs

A und B.

Proof: ( i ) By P 5 , ( A @ B ) , , - ( I @ I ) . ( A @ B ) - ( I . A ) @ ( I . B ) = A , @ B , . (ii) Let the vector s'"' be the m-dimensional summation column vector. Con-

sider m x m matrix A and n x n matrix B. Then

Property P7 is related to Pl( i ) and P2:

P7. (i) u.h = J;(u @ h), ,/br any p x 1 wetors u und h; (ii) at.'.hdr = (u.h)(c.d)' = ild'.h~,'.

Proof: (i) follow5 from P2 with y = 1 after substiiuting u for A and 1) for B; ( i i ) follows from PS, ac1.hd' = (a@c').(h@d') = (u-h)@(c'.d') = (a-h)(c-d)' =

(a-h)(d.c)' = ud'.bcf: alternatively. from P2 and (i) by substituting ac' for A and bd' for B. .

P8. J ; ( A @ c ) = Jb(c @ A) = A(c.)A, /br uny p x y mutrix A und p x 1 vector c..

(See Kollo and Neudecker (1993) for A =I.)

P9. (i) af(h-c) = hl(c.a) = cf(u.h) = u'B,c = h'C,u = c'A,h = Cr= , u, h,c,, where (I: = Ads and A,h = a.h yenerically.

(ii) (A1(BC)), = (Br(C.A)), = (C'(A -B)), = ((A.B)'C),, (iii) t rA1(BC) = t rB1(CA) = trC1(A.B) = tr(A.B)'C.

Proof (i) Trivial. (ii) and (iii) follow from (i) by noting ej A'(B- C)ei = u'(h.c), where a: = A , etc. .

An application of P9(ii) is the following PIO. It arises in the context of Varimax, see Magnus and Neudecker (1991, p 375).

P10. (B'Q), = (C'MC),, where Q = B.MC, C = B.B, M is unjl symmetric matri.~.

Proof: [B1(B. MC)], = [(MC)'(B B)], = [C'MC],, using P9 (ii). . P11. (i) (AA B'),s = (A. B)/, where A is u diagonal mutriv and / = As.

(ii) c = (S. S ) / , \t,here C = S A S ' is the spectrul decomposition of (symmetric ) C, c = C,s.

(See also Rao and Kleffe (1988) for (i). See Horn (1990) for (ii).)

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

T H E HADAMARD PRODUCT 369

Proof: ( i ) By virtue of Pl(i). (2), Pl(ii) and P2, (AABf),s = J' vecAABf = J ' IBOA) vecA = J'IB@A!JAs =!B.AI/ =IA.B)/.

(ii) By using A = B = S in (i). c = C,s = (SAS1),s = (S.S)/. . Pi2. i i j i i . B ~ O / / i i , B > O , o r A ,BsO.

(ii) (AIPA)-(B'ABj 2 ( A ' - B' j f A ( A - B),jiir cinj; p x q iriciiricvs A ciild B mu' ji x p diugoriul mutrices T, A 2 0.

(See Bellman (1970) and Horn (1990) for A, B 2 0 in (i). See Amemiya (1985. p. 205) for A ' - A - fi - fi - B' = B and 1"= A in (ii).)

Prooj: ( i ) follows from P2, as A @ B 2 0. (ii) (A'fA).(B'AB) = J;[(A'PA)@(B'A B)]J,= J;[(A'@ B')(f@A)(A@B)]J,;

(A'.B')PA(A.B) = [J;(A'@B') J,][J;(POA) J,][JJ,(A @ B) J,]. Note that .I,.l;(P@ A) .Ip J;, = J,, J;(PO A) = (PO A) J,J,. Hence r @ A - J,Jk(r@~l)J,Jb=(T@11) - .I, Jb(P@A)-(T@A)J,Jb+ J,Jb (P@A)J,Jb ( I - J P J ) ( f @ ) ( I J P J ; ) 2 0 , a s P , A 2 0 . Thus we estdblish the I-esult. . WP shall now look mtn snme stat~rt~cal prnpertief of Hrrdamard prnd~~ctq of random vectors Some algebraic properties glven before are golng to be used

3. THE EXPECTED VALUES O F RANDOM HADAMARD PRODUCTS

We prove the following result:

Lemma I . The euprcted cu1ur.s of the rundoin Hudur~iurd products x.x und x.y are

( i ) E (s .x ) = cu + p;p,; (ii) E(.uy) = c + p, .p2,

where ti,: = E(x), p,:= E(j),R: = L)(x), V = E(x - p,)(y p , ) ' , to:= R,s and o:= V,s.

Prod. (i) We have E(.ux)= E{J1(.u@x)} = JrE(.u@x) = ~ ' ( v e c R + p , 8 ~ 1 , )

= JrvecR + J r ( p I @ p l ) = to + p, . p i , by virtue of P7(i), (31, Plii) and D(x) = E(sxf ) - E(x) E(xl ) .

(ii) Similarly. . Lemma I(i) can, of course, also be obtained in a more pedestrian way, as E ( x ~ ) = D(ui) +pf = (oii + pz, where toii is the i i t h element of R.

Lemma 2. Lrt x und y he indrprnderit, E(x) = p,, E(y) = p,, E(xxl) = V, and E(yyl) =

V,, then

(i) E(xy'-xy ') = u,t,;; (ii) E(xy'y.xf) = V , . V,;

(iii) E(x.x.y.y) = o;u,. (iv) E(.Y @ x).(y @ y) = vec V, . vec V'.

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

3 70 H. NEUDECKER er al.

Proof: (i) By using P7(ii) and Lemma l(i), E(xy'.xyl) = E(x.x)(y.y)' = E(x.x) E(y.y)' = (o, + p,-pl)(w2 + p2.p2)' = v , v;, as Vi = 4 + pip; hence vi = wi + pi.pi ( i = 1,2), where D(x) = R, and D(y) = R2;

(ii) By using P7(ii), E(xyr.yx') = E(xx'.yyf) = V, V2; (iii) By Lemma l(i), E(x..u.y.y) = (Ex.x).(Ep.y) = v1.o2; (iv) E(x @ x).(y @ y) = vec Vl . vec V2, as E(x @ x) = E vecxx' = vec Ifl, using (3).

In Section 4 we shall derive the variances of x.x and x.y.

4. THE VARIANCES O F RANDOM HADAMARD PRODUCTS

We study the general case first.

Lemma 3. Let p: = E(x), R: = D(x) and o : = R,s, then

Proof: D(x.x)= E(x.x)(x.x)' - E(x.x) E(x.x)'

= E(xxl.xx') -(o + p.p) (o + p.p)', by using P7(ii) and Lemma l(i). H

Lemma 4. Let x and y be independent, E (x) = pl, E(y) = p2, E(xxl) = Vl and E(yyf) = V2, then D(x.y)= V1.V2 -plp;.p2p;.

Proof. D(x.y) = E(x.y)(x-y)' - E(x.y) E(x.y)'= E(xx1.yy') - (p,-p,)(&.p;)

= Vl . V2 - pl pi .,u2 p;, by using P7(ii) and Lemma l(ii). . We now consider the normal case.

Lemma 5. The variance of the Hadamard product xsx, where x - Np(p,R) is 2 R (Q + 2w').

Proof: The basic property to be applied is given in Magnus and Neudecker (1979):

where K,, is the appropriate commutation matrix.

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

THE HADAMARD PRODUCT

It follows that

D(x.x) = JJI,, + KpP) (R O R + f2 @ ,up' + p p 1 0 R ) J p

We used the properties Pl(iv) and P2, keeping in mind the commutativity of the Hadamard product. W

Lemma 6. Let x and y be jointly normally distributed with E(x) = pl, E(y) = p,, E(xx') = V,, E(yyr) = V2, E (xy f ) = V12 and E (yx f ) = V2,, then

Proof: Using P2, Pl(iv) and Theorem 4.3(ii) in Magnus and Neudecker (1979), viz D ( x O . ) = 1/,0 i72 +f ( V 1 2 @ V 2 1 ) - 2 ~ l ~ ' , 8 ~ 1 ~ ; . . After having presented some expectations and variances generally and under normal- ity, we shall especially look into the density functions of x.x.

5. THE DENSITY FUNCTIONS O F THE HADAMARD PRODUCT x.x

A straightforward result is

Lemma 7. Let the density o f x be f(x). Then the density of y = x.x is

where A(x), A(y) and A'I2(y) are diagonal matrices such that A(x)s = x, A(y)s = y and A112(y) > 0 respectively, and xk(y) is the k-th branched inverse transformation from y to x: x1 = A1I2(y)s1,. . . , x~~ = A1I2(y) s ~ ~ , where sk (k = 1 , . . . ,2") are p x 1 vectors with 1 and - 1 arranged lexicographically such that s, = (1,. . . , 1)', . . . , s2, = (- 1,. . . , - 1)'.

Proof: Clearly dy = d(x.x) = 2A(x)dx, where d(.) stands for the differential. Hence axlay'= 1/2AP1(x) . As y = x - x has 2P branched inverse functions xk =A'12(y)sk and corresponding Jacobians Jk = dxk/i?yf and IJkI = 2-PIA-1(x)l = 2-PIA-1'2(y)l, k = 1, ..., 2', so

This establishes the lemma.

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

372 H. NEUDFCKER er ul.

A straightforward specialization to normality is

Lemma 8. I f .u hus multiuuriute normul density with p = 0 and R = [Id , vlr (271) ""exp( - I j2.~'Q; Is) , then J = x.x hu.5 den.>ity Junction (271) ' '"IA 1 ' 2 ( s ) l cxp( - !/2 ;,'<i;), .,i,!y7c <i; = Qd 5.

Proof: By Lemma 7 and noting that u ;Qi l u, = y'to, k = 1, . . . .2". . The next step is to find the moment-generating function of Y . Y under normality

6. T H E MOMENT-GENERATING FUNCTION O F T H E HADAMARD PRODUCT x.x UNDER NORMALITY

The ~i~oment-generati~lg function of .Y-x is defined as rn(t) = Eexp t l ( s .x ) , t being a vector variable. Its domain of definition will implicitly be given below. We shall prove the following result:

Lemma 9. The moment-ycwcratinyrain function u f ' s . ~ is

where W ' :=Q ' -2A(t)und W>O.

Proof: We write t ' (x.x) = u'Alt).u and use the result (due to Heijmans. see Neudecker ( 1990)):

where W ': = R ' - 2 ~ ~ 4 . 7 such that W > O and A' = -4. Replacing TA by A(t) in (9) and subsequently transforming prA(t)p and p'A(t) WA(t)p

yields the result. The detailed algebra goes as follows.

pfA(t )p = fl(p.p), by virtue of P9(i);

and

plA(t) w A ( t ) p = t rA(t )w'A(t ) W = j v e ~ A ( t ) ) - ' ( W @ ~ p ' ) vec A(t) = t' J ;(W@ppl) J,t = tr(W.pp')t , by virtue of ( I ) , (2), Pl(ii) and P2. . [ I ] Amemiya, 7. ( 1985). Arlwnixd Eco~~omerric,.s. Basil Blackwell, Oxford. 121 Bellman, R. (1970). Introduction t o M u r r i ~ Anulysis ( 2 ~ d ed). McGraw-Hill. New York. 131 Browne. M. W . (1974). Generalired least squares estimators in the analysis of covariance structures.

S. Afr. Srurrsr. J . 8. 1 24. [4] Horn, K. G . (1990). The Hadamard product. Prot. Sjtiip. Appl. Murh. 40. $7- 169 1-51 Kollo, T. and Neudecker, H. (1993). Asymptotics ofeigenvalues and unit-lengthe~genvectors of sample

variance and correlation matrices. J. Mulrr. A J ~ 47, 2 8 3 3 0 0 .

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3

THE HADAMARD PRODUCT 373

[6] Magnus, J. R. [1988]. Linear Structures. Griffin's Statistical Monographs, 42. Charles Griffin and Company Ltd, London and Oxford University Press. New York.

[7] Magnus, J. R. and Neudecker. H. (1979). The commutation matrix: Some properties and applications. Ann. Stutist. 7. 381-394.

C8-j Magnus. J. R.. and Neudecker. H. (1991). M u t r i . ~ Diferential Calculus with Appliccirions in Statistics and Econometrics (2nd ed). Wiley, Chichester.

[9] Neudecker. H. (1975). A derivation of the Hessian of the (concentrated) iikelihood function of the factor model employing the Schur product. Br. J. Math. Statist. P s ~ c h o l . 28. 152- 156.

[lo] Neudecker. H. (1981). On the matrix formulation of Kaiser's varimax criterion. Psjchometrika 46, 343 345

[l 11 Neudccker, H. (1983). On Jacobians oftransformations with skew-symmetric. strictly (lower) triangular or diagonal matrix arguments. Linear Multi. Alg. 14.271-295.

[I21 Neudecker, H. (1990). The variance matrix of a matrix quadratic form under normality assumptions. A derivation based on its moment-generating function. Statistics 21,455-459.

[13] Neudecker, H. (1993). The Hessian matrixjor Imuqe Factor Analysis. in: Steyer, R., Wender, K. F. and Widaman, K. F. (eds.) Psychometric Methodology. Gustav Fischer, Stuttgart. 361-365.

[14] Neudecker. H. and Wansbeek. T. (1983). Some results on commutation matrices, with statistical applications. Can. J. Statist. 11,221-231.

[I51 Pukelsheim, F. (1977). On Hsu's model in regression analysis. Statistics 8. 323-331. [16] Rao, C. R. and Kleffe, J. (1988). Estimation vf Variance Components and Applications. North-Holland.

Amsterdam. [17] Styan. G. P. H. (1973). Hadamard products and multivariate statistical analysis. Linear Alq. Appl. 6,

217-240.

H. Neudecker. S. Liu lnstitute of Actuarial Science and Econometrics. University of Amsterdam Roetersstraat 11 NL-1018 WB Amsterdam

W. Polasck lnstitute of Statistics and Econometrics University of Basel Petersgraben 51 CH-4051 Basel

Dow

nloa

ded

by [

Uni

vers

itaet

sbib

lioth

ek D

ortm

und]

at 0

9:07

15

July

201

3