arxiv:1511.09064v1 [math.rt] 29 nov 2015matrix coefficients for unitary representations of nilpote...

21
arXiv:1511.09064v1 [math.RT] 29 Nov 2015 Stepwise Square Integrable Representations: the Concept and Some Consequences Joseph A. Wolf Abstract There are some new developments on Plancherel formula and growth of matrix coefficients for unitary representations of nilpotent Lie groups. These have several consequences for the geometry of weakly symmetric spaces and analysis on parabolic subgroups of real semisimple Lie groups, and to (infinite dimensional) locally nilpotent Lie groups. Many of these consequences are still under develop- ment. In this note I’ll survey a few of these new aspects of representation theory for nilpotent Lie groups and parabolic subgroups. 1. Introduction. There is a well developed theory of square integrable representations of nilpotent Lie groups [17]. It is based on the general representation theory of Kirillov [12] for connected nilpotent real Lie groups. A connected simply connected Lie group N with center Z is called square integrable if it has unitary representations π whose coefficients f u,v (x)= u, π (x)vsatisfy | f u,v |∈ L 2 (N/Z). If N has one such square integrable representation then there is a certain polynomial function P(γ ) on the linear dual space z of the Lie algebra of Z that is key to harmonic analysis on N. Here P(γ ) is the Pfaffian of the antisymmetric bilinear form on n/z given by b λ (x, y)= λ ([x, y]) where γ = λ | z . The square integrable representations of N are certain easily–constructed representations π γ where γ z with P(γ ) = 0, Plancherel almost irreducible unitary representations of N are square integrable, and up to an explicit constant |P(γ )| is the Plancherel density of the unitary dual N at π λ . This theory has some interesting analytic consequences [26]. More recently there was a serious extension of that theory [28]. Under certain conditions, the nilpotent Lie group N has a decomposition into subgroups that have Joseph A. Wolf Department of Mathematics, University of California, Berkeley, CA 94720–3840, USA, e-mail: [email protected] 1

Upload: others

Post on 08-Oct-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

arX

iv:1

511.

0906

4v1

[mat

h.R

T]

29 N

ov 2

015

Stepwise Square Integrable Representations:the Concept and Some Consequences

Joseph A. Wolf

Abstract There are some new developments on Plancherel formula and growth ofmatrix coefficients for unitary representations of nilpotent Lie groups. These haveseveral consequences for the geometry of weakly symmetric spaces and analysison parabolic subgroups of real semisimple Lie groups, and to(infinite dimensional)locally nilpotent Lie groups. Many of these consequences are still under develop-ment. In this note I’ll survey a few of these new aspects of representation theory fornilpotent Lie groups and parabolic subgroups.

1. Introduction.

There is a well developed theory of square integrable representations of nilpotentLie groups [17]. It is based on the general representation theory of Kirillov [12] forconnected nilpotent real Lie groups. A connected simply connected Lie groupNwith centerZ is calledsquare integrableif it has unitary representationsπ whosecoefficientsfu,v(x) = 〈u,π(x)v〉 satisfy| fu,v| ∈ L 2(N/Z). If N has one such squareintegrable representation then there is a certain polynomial functionP(γ) on thelinear dual spacez∗ of the Lie algebra ofZ that is key to harmonic analysis onN. HereP(γ) is the Pfaffian of the antisymmetric bilinear form onn/z given bybλ (x,y) = λ ([x,y]) whereγ = λ |z . The square integrable representations ofN arecertain easily–constructed representationsπγ whereγ ∈ z∗ with P(γ) 6= 0, Plancherelalmost irreducible unitary representations ofN are square integrable, and up to anexplicit constant|P(γ)| is the Plancherel density of the unitary dualN at πλ . Thistheory has some interesting analytic consequences [26].

More recently there was a serious extension of that theory [28]. Under certainconditions, the nilpotent Lie groupN has a decomposition into subgroups that have

Joseph A. WolfDepartment of Mathematics, University of California, Berkeley, CA 94720–3840, USA, e-mail:[email protected]

1

Page 2: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

2 Joseph A. Wolf

square integrable representations, and the Plancherel formula then is synthesizedexplicitly in terms of the Plancherel formulae of those subgroups. In particular theextended theory applies to nilradicals of minimal parabolic subgroups [28]. With aminor technical adjustment it has just been extended to nilradicals of arbitrary realparabolics [32]. The consequences include explicit Plancherel and Fourier inversionformulas. Applications include analysis on minimal parabolic subgroups [29] and,more generally, on maximal amenable subgroups of parabolics [32], They also in-clude analysis on commutative spaces, i.e. on Gelfand pairs[31]. We sketch someof these developments. Due to constraints of time and space we pass over many as-pects of operator theory and orbit geometry, for example those described in [2], [3]and [4], related to stepwise square integrable representations.

In Section 2 we recall the basic facts [17], with a few extensions, on squareintegrable representations of nilpotent Lie groups. In Section 3 we recall the conceptand main results for stepwise square integrable nilpotent Lie group.

In Section 4 we show how nilradicals of minimal parabolic subgroups havethe required decomposition for stepwise square integrability. This is a constructionbased on concept of strongly orthogonal restricted roots.

In Section 5 we indicate the consequences for homogeneous compact nilman-ifolds, and in Section 6 we mention the application to analysis on commutativenilmanifolds.

In Section 7 we start the extension of stepwise square integrability results fromthe nilradicalN of a minimal parabolicP= MAN to various subgroups that containN. This section concentrates on the subgroupMN and takes advantage of principalorbit theory. That gives a sharp simplification to the Plancherel and Fourier Inversionformulae. In Section 8 we look atP and its subgroupAN. They are not unimodular,so we introduce the Dixmier-Pukanszky operatorD whose semi–invariance balancesthat of the modular function. It is a key point for the Plancherel and Fourier Inversionformulae.

Sections 9 and 10 are a short discussion of work in progress onthe extensionof results from minimal parabolics to parabolics in general. There are two placeswhere matters diverge from the minimal parabolic case. First, there is a technicaladjustment to the definition of stepwise square integrable representation, causedby the fact that in the non–minimal case the restricted rootsneed not form a rootsystem. Second, again for technical reasons, the explicit Plancherel Formula onlycomes through for the maximal amenable subgroupsUAN of G, and not for all ofthe parabolic.

This work was partially supported by a Simons Foundation grant and by theaward of a Dickson Emeriti Professorship. It expands a talk at the 11-th InternationalWorkshop “Lie Theory and Its Applications in Physics” in Varna. My thanks to Prof.Vladimir Dobrev and the others on the organizing committee for hospitality at thatWorkshop.

Page 3: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 3

2. Square Integrable Representations.

Let G be a unimodular locally compact group with centerZ, and let π be anirreducible unitary representation. We associate the central characterχπ ∈ Z byπ(z) = χπ(x) ·1 for z∈ Z. Consider a matrix coefficientfu,v : x 7→ 〈u,π(x)v〉. Then| fu,v| is a well defined function onG/Z. Fix Haar measuresµG on G, µZ on Z andµG/Z onG/Z such thatdµG = dµZ dµG/Z . The following results are well known.

Theorem 2.1.The following conditions onπ ∈ G are equivalent.(1) There exist nonzero u,v∈ Hπ with | fu,v| ∈ L 2(G/Z).(2) | fu,v| ∈ L 2(G/Z) for all u,v∈ Hπ .(3) π is a discrete summand of the representationIndG

Z (χπ).

Theorem 2.2.If the conditions ofTheorem 2.1are satisfied for an irreducibleπ ∈G, then there is a numberdegπ > 0 such that

G/Zfu,v(x) fu′ ,v′(x)dµG/Z(xZ) = 1

degπ 〈u,u′〉〈vv′〉 (1)

for all u,u′,v,v′ ∈ Hπ . If π1,π2 ∈ G are inequivalent and satisfy the conditions ofTheorem 2.1, andχπ1 = χπ2, then

G/Z〈u,π1(x)v〉〈u′,π2(x)v′〉dµG/Z(xZ) = 0 (2)

for all u,v∈ Hπ1 and all u′,v′ ∈ Hπ2 .

The main results of [17] shows exactly how this works for nilpotent Lie groups:

Theorem 2.3.Let N be a connected simply connected Lie group with center Z,n

and z their Lie algebras, andn∗ the linear dual space ofn. Let λ ∈ n∗ and letπλdenote the irreducible unitary representation attached toAd∗(N)λ by the Kirillovtheory[12]. Then the following conditions are equivalent.

(1) πλ satisfies the conditions ofTheorem 2.1.(2) The coadjoint orbitAd∗(N)λ = {ν ∈ n∗ | ν|z = λ |z.(3) The bilinear form bλ (x,y) = λ ([x,y]) onn/z is nondegenerate.(4) The universal enveloping algebraU (z) is the center ofU (n).

The Pfaffian polynomialPf(bλ ) is a polynomial function P(λ |z) on z∗, and the setof representationsπλ for which these conditions hold, is parameterized by the set{γ ∈ z∗ | P(γ) 6= 0} (which is empty or Zariski open inz∗).

We will say that the connected simply connected Lie groupN is square integrableif there existsλ ∈ n∗ such thatP(λ |z) 6= 0}. For convenience we will sometimeswrite P(λ ) for P(λ |z) andπγ for πλ whereγ = λ |z .

Theorem 2.4.Let N be a square integrable connected simply connected Lie groupwith center Z. Then Plancherel measure onN is concentrated on{πλ | P(λ ) 6= 0},and there the Plancherel measure is given by the measure|P(λ )dλ | on z∗ and theformal degreedegπλ = |P(λ |z)|.

Page 4: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

4 Joseph A. Wolf

Givenγ ∈ z∗ with P(γ) 6=0 and a Schwartz class (C (N)) function f onN we writeO(γ) for the co-adjoint orbit Ad∗(N)γ = γ + z⊥ , fγ for the restriction off ·exp toO(γ), and fγ for the Fourier transform offγ onO(γ).

Theorem 2.5.Let N be a square integrable connected simply connected Lie groupwith center Z and f∈ C (N). If γ ∈ z∗ with P(γ) 6= 0 then the distribution characterof πγ is given by

Θπγ ( f ) = trace∫

Nf (x)πγ (x)dµG(x) = c−1|P(γ)|−1

ν∈O(γ)fγ dν (3)

where c= d!2d and d= dim(n/z)/2 and dν is ordinary Lebesgue measure on theaffine spaceO(γ) . The Fourier Inversion formula for N is

f (x) = c∫

z∗Θγ(rx f )|P(γ)|dγ where(rx f )(y) = f (yx) (right translate). (4)

There also are multiplicity results onL 2(N/Γ ) whereN is square integrable andΓ is a discrete co-compact subgroup, but they are the same as inthe stepwise squareintegrable case, so we postpone their description.

3. Stepwise Square Integrability.

In order to to go beyond square integrable nilpotent groups,we suppose that theconnected simply connected nilpotent Lie group decomposesas

N =L1L2 . . .Lm−1Lm where

(a) eachLr has unitary representations with coeff inL2(Lr/Zr),

(b) Nr := L1L2 . . .Lr is normal inN with Nr = Nr−1⋊Lr ,

(c) [lr ,zs] = 0 and[lr , ls]⊂ v for r > swith lr = zr +vr

wheren= s+v, s=⊕zr andv=⊕vr .

(5)

We will use the following notation.

(a)dr =12 dim(lr/zr) so 1

2 dim(n/s) = d1+ · · ·+dm,

andc= 2d1+···+dmd1!d2! . . .dm!

(b) bλr : (x,y) 7→ λ ([x,y]) viewed as a bilinear form onlr/zr

(c) S= Z1Z2 . . .Zm = Z1×·· ·×Zm whereZr is the center ofLr

(d) P : polynomialP(λ ) = Pf(bλ1)Pf(bλ2

) . . .Pf(bλm) on s∗

(e) t∗ = {λ ∈ s∗ | P(λ ) 6= 0}

(f) πλ ∈ N for λ ∈ s∗ with P(λ ) 6= 0, irreducible unitary representation

of N = L1L2 . . .Lm constructed as follows.

(6)

Page 5: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 5

Start with the representationπλ1∈ N1 specified byλ1 ∈ z∗1 with Pf(bλ1

) 6= 0.Choose an invariant polarizationp′1 ⊂ n2 for the linear functionalλ ′

1 ∈ n∗2 that agreeswith λ1 on n1 and vanishes onl2. SinceLr centralizesSr−1, ad∗(l2)(λ ′

1)|z1+l2 = 0,so p′1 = p1 + l2 wherep1 is an invariant polarization forλ1 ∈ n∗1. The associatedrepresentations areπ ′

λ1∈ N2 andπλ1

∈ N1. Note thatN2/P′1 = N1/P1 , so the rep-

resentation spacesHπ ′λ1

= L 2(N2/P′1) = L 2(N1/P1) = Hπλ1

. In other words,π ′λ1

extendsπλ1to a unitary representation ofN2 on the same Hilbert spaceHπλ1

, anddπλ ′

1(z2) = 0. Now the Mackey Little Group method gives us

Lemma 3.1.The irreducible unitary representations of N2, whose restrictions to N1are multiples ofπλ1

, are theπ ′λ1⊗γ whereγ ∈ L2 = N2/N1 .

Givenλ2 ∈ z∗2 with Pf(bλ2) 6= 0 we haveπλ2

∈ L2 with coefficients inL 2(L2/Z2).In the notation of Lemma 3.1 we define

πλ1+λ2∈ N2 by πλ1+λ2

= π ′λ1⊗πλ2

. (7)

Proposition 3.2.The coefficients fz,w(xy) = 〈z,πλ1+λ2(xy)w〉 of πλ1+λ2

belong to

L 2(N2/S2), in fact satisfy|| fz,w||2L 2(Nr/Sr )= ||z||2||w||2

deg(πλ1)...deg(πλr )

.

Proposition 3.2 starts a recursion usingNr+1 = Nr ⋊Lr+1. We fix nonzeroλi ∈z∗i for 1 ≦ i ≦ r + 1, and we start with the representationπλ1+···+λr constructed

step by step from the square integrable representationsπλi∈ Li for 1 ≦ i ≦ r. The

representation spaceHπλ1+···+λr= Hπλ1

⊗ . . . ⊗Hπλr. The coefficients ofπλ1+···+λr

have absolute value inL 2(Nr/Sr). They satisfy

|| fz,w||2L 2(Nr/Sr )

= ||z||2||w||2

deg(πλ1)...deg(πλr )

. (8)

Thenπλ1+···+λr extends to a representationπ ′λ1+···+λr

of Lr+1 on the same HilbertspaceHπλ1+···+λr

, and it satisfiesdπ ′λ1+···+λr

(zr+1) = 0. As in Lemma 3.1,

Lemma 3.3.The irreduciblesπ ∈ Nr+1, whose restrictions to Nr are multiples of

πλ1+···+λr , are theπ ′λ1+···+λr

⊗γ whereγ ∈ Lr+1 = Nr+1/Nr .

As in Proposition 3.2, defineπλ1+···+λr+1= π ′

λ1+···+λr⊗πλr+1

. Then

Proposition 3.4.The coefficients fz,w(x1 . . .xr+1) = 〈z,πλ1+···+λr+1(x1x2 · · ·xr+1)w〉

of πλ1+···+λr+1belong toL 2(Nr+1/Sr+1), in fact satisfy

|| fz,w||2L 2(Nr+1/Sr+1)

= ||z||2||w||2

deg(πλ1)...deg(πλr+1

) .

Since degπλr = |Pf(bλr )|, Proposition 3.4 is the recursion step for our construc-tion. Passing to the end caser +1= m we see that Plancherel measure is concen-trated on{πλ | λ ∈ t∗}. Using (5)(c) to see that conjugation by elements ofLs hasno effect on the Pf(bλr ) for r < s, we arrive at

Page 6: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

6 Joseph A. Wolf

Theorem 3.5.Let N be a connected simply connected nilpotent Lie group that satis-fies(5). Then Plancherel measure for N is concentrated on{πλ | λ ∈ t∗,P(λ ) 6= 0}.If λ ∈ t∗, P(λ ) 6= 0 and u,v∈Hπλ , then the coefficient fu,v(x) = 〈u,πν(x)v〉 satisfies

|| fu,v||2L 2(N/S) = ||u||2||v||2/|P(λ )| . (9)

The distribution characterΘπλ : f 7→ trace∫

G f (x)π(x)dx ofπλ is given by

Θπλ ( f ) = c−1|P(λ )|−1∫

O(λ )fλ (ξ )dνλ (ξ ) for f ∈ C (N) (10)

where C (N) is the Schwartz space,O(λ ) = Ad∗(N)λ = s⊥ + λ , fλ is the liftfλ (ξ ) = f (exp(ξ )), fλ is its classical Fourier transform, and dνλ is the translateof normalized Lebesgue measure froms⊥ to Ad∗(N)λ . Further,

f (x) = c∫

t∗Θπλ (rx f )|P(λ )|dλ for f ∈ C (N). (11)

Definition 3.6. The representationsπλ of (6(f)) are thestepwise square integrablerepresentations ofN relative to (5). ♦

The left action(l(x) f )(g) = f (x−1g) and the right action(r(y) f )(g) = f (gy) of Non functions carries over to coefficients ofπ asl(x)r(y) fu,v = fπ(x)u,π(y)v. If π = πλstepwise square integrable,u,v ∈ Hπλ areC∞ vectors, and ifΦ andΨ belong tothe universal enveloping algebraU (n), thenl(Φ)r(Ψ ) fu,v = fdπ(Ψ )u,dπ(Φ)v is just

another coefficient,C∞ andL 2(N/S). If ζλ ∈ S is the quasicentral character ofπλit follows that fu,v belongs to the relative Schwartz spaceC (N/S,ζλ ). In particularit follows that | fu,v| ∈ L p(N/S) for all p≧ 1. Taking Schwartz class wave packetsoverSof coefficient functions of stepwise square integrable representations ofN onecan express the Plancherel formula of Theorem 3.5 in terms ofcoefficient functions.

4. Nilradicals of Minimal Parabolics.

Fix a real simple Lie groupG, an Iwasawa decompositionG= KAN, and a minimalparabolic subgroupQ = MAN in G. Let m= rankRG = dimRA . As usual, writekfor the Lie algebra ofK, a for the Lie algebra ofA, andn for the Lie algebra ofN.Completea to a Cartan subalgebrah of g. Thenh = t+ a with t = h∩ k. Now wehave root systems

∆(gC,hC): roots ofgC relative tohC (ordinary roots),

∆(g,a): roots ofg relative toa (restricted roots),

∆0(g,a) = {α ∈ ∆(g,a) | 2α /∈ ∆(g,a)} (nonmultipliable).

(12)

Page 7: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 7

Here ∆(g,a) and ∆0(g,a) are root systems in the usual sense. Any positive rootsystem∆+(gC,hC)⊂ ∆(gC,hC) defines positive systems

∆+(g,a) = {γ|a | γ ∈ ∆+(gC,hC) andγ|a 6= 0},

∆+0 (g,a) = ∆0(g,a)∩∆+(g,a).

(13)

We can (and do) choose∆+(g,h) so that

n is the sum of the positive restricted root spaces and

if γ ∈ ∆(gC,hC) andγ|a ∈ ∆+(g,a) thenγ ∈ ∆+(gC,hC).(14)

Two roots are calledstrongly orthogonalif their sum and their difference are notroots. Then they are orthogonal. The Kostant cascade construction is

β1 ∈ ∆+(g,a) is a maximal positive restricted root and

βr+1 ∈ ∆+(g,a) is a maximum among the roots of∆+(g,a)

that are orthogonal to allβi with i ≦ r

(15)

Then theβr are mutually strongly orthogonal. Eachβr ∈ ∆+0 (g,a), andβ1 is unique

because∆(g,a) is irreducible. For 1≦ r ≦ m define

∆+1 = {α ∈ ∆+(g,a) | β1−α ∈ ∆+(g,a)} and

∆+r+1 = {α ∈ ∆+(g,a)\ (∆+

1 ∪·· ·∪∆+r ) | βr+1−α ∈ ∆+(g,a)}.

(16)

Lemma 4.1.If α ∈ ∆+(g,a), eitherα ∈ {β1, . . . ,βm} or α belongs to just one∆+r .

Lemma 4.2.∆+r ∪{βr}= {α ∈ ∆+ | α ⊥ βi for i < r and 〈α,βr〉> 0}. In particu-

lar, [lr , ls]⊂ lt where t= min{r,s}.

Lemma 4.1 shows that the Lie algebran of N is the direct sum of its subspaces

lr = gβr +∑∆+rgα for 1≦ r ≦ m (17)

and Lemma 4.2 shows thatn has an increasing foliation by ideals

nr = l1+ l2+ · · ·+ lr for 1≦ r ≦ m. (18)

Now we will see that the corresponding group level decomposition N = L1L2 . . .Lm

and the semidirect product decompositionsNr = Nr−1⋊Lr satisfy (5). Denote

sβr is the Weyl group reflection inβr and

σr : ∆(g,a)→ ∆(g,a) by σr(α) =−sβr (α).(19)

Note thatσr(βs) =−βs for s 6= r, +βs if s= r. If α ∈ ∆+r we still haveσr(α)⊥ βi

for i < r and 〈σr(α),βr〉 > 0. If σr(α) < 0 thenβr − σr(α) > βr contradictingmaximality ofβr . Thus, using Lemma 4.2,σr(∆+

r ) = ∆+r .

Page 8: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

8 Joseph A. Wolf

Lemma 4.3.If α ∈ ∆+r thenα +σr(α) = βr . (It is possible thatα = σr(α) = 1

2βr

when12βr is a root.). If α,α ′ ∈ ∆+

r andα +α ′ ∈ ∆(g,a) thenα +α ′ = βr .

Lemma 4.4.Let n be a nilpotent Lie algebra,z its center, andv a vector spacecomplement toz in n. Suppose thatv= u+u′, u= ∑ua andu′ = ∑u′a , andz= ∑ zb

with dimzb = 1 in such a way that(i) each[ua,ua] = 0= [u′a,u′a], (ii) if a1 6= a2 then

[ua1,u′a2] = 0 and(iii) for each a there is a nondegenerate pairingua⊗u′a → zba , by

u⊗u′ 7→ [u,u′]. Thenn is a direct sum of Heisenberg algebraszba +ua+u′a and thecommutative algebra that is the sum of the remainingzb .

Now one runs through a number of special situations: (1) Ifg is the split realform of gC then eachLr has square integrable representations. (2) Ifg is simple butnot absolutely simple then eachLr has square integrable representations. (3) IfGis the quaternion special linear groupSL(n;H) thenL1 has square integrable repre-sentations. (4) IfG is the groupE6,F4 of collineations of the Cayley projective planethenL1 has square integrable representations. (5) The groupL1 has square integrablerepresentations. (6) Ifg is absolutely simple then eachLr has square integrable rep-resentations. Putting these together, Theorem 3.5 appliesto nilradicals of minimalparabolic subgroups:

Theorem 4.5.Let G be a real reductive Lie group, G= KAN an Iwasawa de-composition,lr andnr the subalgebras ofn defined in(17) and (18), and Lr andNr the corresponding analytic subgroups of N. Then the Lr and Nr satisfy(5). Inparticular, Plancherel measure for N is concentrated on{πλ | λ ∈ t∗}. If λ ∈ t∗,and if u and v belong to the representation spaceHπλ of πλ , then the coefficient

fu,v(x) = 〈u,πλ (x)v〉 satisfies|| fu,v||2L 2(N/S)= ||u||2||v||2

|P(λ )| . The distribution character

Θπλ of πλ satisfiesΘπλ ( f ) = c−1|P(λ )|−1∫O(λ ) fλ (ξ )dνλ (ξ ) for f ∈ C (N). Here

C (N) is the Schwartz space,O(λ ) is the coadjoint orbitAd∗(N)λ = s⊥ + λ , fγis the lift fγ (ξ ) = f (exp(ξ )) to s⊥+λ , fγ is its classical Fourier transform , anddνλ is the translate of normalized Lebesgue measure froms⊥ to Ad∗(N)λ . ThePlancherel formula on N is f(x) = c

∫t∗ Θπλ (rx f )|P(λ )|dλ for f ∈ C (N).

5. Compact Nilmanifolds.

Here are the basic facts on discrete uniform (i.e. co-compact) subgroups of con-nected simply connected nilpotent Lie groups. See [21, Chapter 2] for an exposition.

Proposition 5.1.The following are equivalent.

• N has a discrete subgroupΓ with N/Γ compact.• N ∼= NR where NR is the group of real points in a unipotent linear algebraic

group defined over the rational number fieldQ• n has a basis{ξ j} for which the coefficients ck

i, j in [ξi ,ξ j ] = ∑cki, jξk are rational

numbers.

Page 9: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 9

Under those conditions letnQ

denote the rational span of{ξ j} and letnZ

be the in-tegral span. Thenexp(n

Z) generates a discrete subgroup NZ of N= NR and NR/NZ

is compact. Conversely, ifΓ is a discrete co-compact subgroup of N then theZ–spanof exp−1(Γ ) is a lattice inn for which any generating set{ξ j} is a basis ofn suchthat the coefficients ck

i, j in [ξi ,ξ j ] = ∑cki, jξk are rational numbers.

The conditions of Proposition 5.1 hold for the nilpotent groups studied in Section4; there one can choose the basis{ξ j} of n so that theck

i, j are integers.

The basic facts on square integrable representations that occur in compact quo-tientsN/Γ , as described in [17, Theorem 7], are

Proposition 5.2.Let N be a connected simply connected nilpotent Lie group thathas square integrable representations, and letΓ a discrete co-compact subgroup.Let Z be the center of N and normalize the volume form onn/z by normalizing Haarmeasure on N so that N/ZΓ has volume1. Let P be the corresponding Pfaffianpolynomial onz∗. Note thatΓ ∩Z is a lattice in Z andexp−1(Γ ∩Z) is a lattice(denote itΛ ) in z. That defines the dual latticeΛ∗ in z∗. Then a square integrablerepresentationπλ occurs inL 2(N/Γ ) if and only if λ ∈ Λ∗, and in that caseπλoccurs with multiplicity|P(λ )|.Definition 5.3. Let N = NR be defined overQ as in Proposition 5.1, so we have afixed rational formNQ. We say that a connected Lie subgroupL ⊂ N is rational ifL∩NQ is a rational form ofL, in other words ifl∩n

Qcontains a basis ofl. We say

that a decomposition (5) isrational if the subgroupsLr andNr are rational. ♦

The following is immediate from this definition.

Lemma 5.4.Let N be defined overQ as inProposition 5.1with rational structuredefined by a discrete co-compact subgroupΓ . If the decomposition(5) is rationalthen eachΓ ∩Zr in Zr , eachΓ ∩Lr in Lr , eachΓ ∩Sr in Sr , and eachΓ ∩Nr in Nr ,is a discrete co-compact subgroup defining the same rationalstructure as the onedefined by its intersection with NQ .

Now assume thatN andΓ satisfy the rationality conditions of Lemma 5.4. Thenfor eachr, Zr ∩Γ is a lattice in the centerZr of Lr , andΛr := log(Zr ∩Γ ) is alattice in its Lie algebrazr . That defines the dual latticeΛ∗

r in z∗r . We normalize thePfaffian polynomials on thez∗r , and thus the polynomialP on s∗, by requiring thattheNr/(Sr · (Nr ∩Γ )) have volume 1.

Theorem 5.5.Letλ ∈ t∗. Then a stepwise square integrable representationπλ of Noccurs inL 2(N/Γ ) if and only if eachλr ∈ Λ∗

r , and in that case the multiplicity ofπλ onL 2(N/Γ ) is |P(λ )|.

6. Commutative Spaces.

A commutative spaceX = G/K, or equivalently a Gelfand pair(G,K), consists of alocally compact groupG and a compact subgroupK such that the convolution alge-braL 1(K\G/K) is commutative. WhenG is a connected Lie group it is equivalent

Page 10: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

10 Joseph A. Wolf

to say that the algebraD(G,K) of G–invariant differential operators onG/K is com-mutative. We say that the commutative spaceG/K is a commutative nilmanifold ifit is a nilmanifold in the sense that some nilpotent analyticsubgroupN of G actstransitively. WhenG/K is connected and simply connected it follows thatN is thenilradical ofG, thatN acts simply transitively onG/K, and thatG is the semidirectproduct groupN⋊K, so thatG/K = (N⋊K)/K. In this section we study the role ofsquare integrability and stepwise square integrability for commutative nilmanifoldsG/K = (N⋊K)/K.

The cases whereG/K and(G,K) areirreducible in the sense that[n,n] (whichmust be central) is the center ofn andK acts irreducibly onn/[n,n], have beenclassified by E. B. Vinberg ([22], [23]). See [26,§13.4B] for the Lie algebra structurev× v → z. The classification of commutative nilmanifolds is based onVinberg’swork and was completed by O. Yakimova in [34] and [35].

It turns out that almost all commutative manifolds correspond to nilpotent groupsthat are square integrable. The exceptions are those with a certain direct factor, andin those cases the nilpotent group is stepwise square integrable in two steps, so inthose cases the Plancherel formula follows directly from the general result above.See [31] for the details.

7. Minimal Parabolics: Subgroup MN.

Fix an Iwasawa decompositionG= KAN for a simple Lie groupG and the minimalparabolic subgroupQ = MAN. As usual, writek for the Lie algebra ofK, a forthe Lie algebra ofA, m for the Lie algebra ofM, andn for the Lie algebra ofN.Completea to a Cartan subalgebrah of g. Then we have root systems∆(gC,hC),∆(g,a) and∆0(g,a) described in (12).M is the centralizer ofA in K. Write 0 foridentity component; thenQ0 = M0AN.

Recall the Pf–nonsingular sett∗ = {λ ∈ s∗ |Pf(bλ ) 6= 0} of (6e); so Ad∗(M)t∗ =t∗. Further, ifλ ∈ t∗ andc 6= 0 thencλ ∈ t∗, in fact Pf(bcλ ) = cdim(n/s)/2Pf(bλ ).

Fix anM–invariant inner product(µ ,ν) ons∗ . So Ad∗(M) preserves each spheres∗t = {λ ∈ s∗ | (λ ,λ ) = t2}. Two orbits Ad∗(M)µ and Ad∗(M)ν are of thesame orbittype if the isotropy subgroupsMµ andMν are conjugate, and an orbit isprincipalifall nearby orbits are of the same type. SinceM ands∗t are compact, there are onlyfinitely many orbit types ofM on s∗t , there is only one principal orbit type, and theunion of the principal orbits forms a dense open subset ofs∗t whose complementhas codimension≧ 2. See [5, Chapter 4, Section 3] for a complete treatment of thismaterial, or [10, Part II, Chapter 3, Section 1] modulo references to [5], or [18, Cap.5] for a basic treatment, still with some references to [5].

The action ofM on s∗ commutes with dilation so the structural results on thest

also hold ons∗ =⋃

t>0s∗t . Define the Pf-nonsingular principal orbit set as follows:

Page 11: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 11

u∗ = {λ ∈ t∗ | Ad∗(M)λ is a principalM-orbit ons∗}. (20)

Now principal orbit setu∗ is a dense open set with complement of codimension≧ 2 in s∗. If λ ∈ u∗ andc 6= 0 thencλ ∈ u∗ with isotropyMcλ = Mλ . If λ ∈ u∗t :=u∗∩s∗t , so Ad∗(M)λ is a Pf-nonsingular principal orbit ofM on the spheres∗t , thenAd∗(M0)λ is a principal orbit ofM0 on s∗t . Principal orbit isotropy subgroups ofcompact connected linear groups are studied in [11] and the possibilities for theisotropy(M0)λ are essentially known. The following lets us go from(M0)λ to Mλ .

Proposition 7.1.([29]) Suppose that G is connected and linear. Then M= FZGM0

where ZG is the center of G, F= (exp(ia)∩K) is an elementary abelian2–group,and Ad∗(F) acts trivially ons∗. If λ ∈ u∗ then the isotropy Mλ = FZG(M0)λ .

Thus the groupsMλ are specified by the work of W.–C. and W.–Y. Hsiang [11].

Given λ ∈ u∗ the stepwise square integrable representationπλ ∈ N one provesthat the Mackey obstructionε ∈ H2(Mλ ;U(1)) is trivial, and in fact thatπλ extendsto a unitary representationπ†

λ of N⋊Mλ on the representation space ofπλ .

Eachλ ∈ u∗ now defines classes

E (λ ) :={

π†λ ⊗ γ | γ ∈ Mλ

}, F (λ ) :=

{IndNM

NMλ(π†

λ ⊗ γ) | π†λ ⊗ γ ∈ E (λ )

}(21)

of irreducible unitary representations ofN⋊Mλ andNM. The Mackey little groupmethod, plus the fact that the Plancherel density onN is polynomial ons∗ , ands∗ \u∗ has measure 0 int∗, gives us

Proposition 7.2.Plancherel measure for NM is concentrated on⋃

λ∈u∗ F (λ ), equiv-alence classes of irreducible representationsηλ ,γ := IndNM

NMλ(π†

λ ⊗ γ) such that

π†λ ⊗ γ ∈ E (λ ) andλ ∈ u∗. Further

ηλ ,γ |N =(

IndNMNMλ

(π†λ ⊗ γ)

)∣∣∣N=

M/Mλ(dimγ)πAd∗(m)λ d(mMλ ).

There is a Borel sectionσ to u∗ → u∗/Ad∗(M) that picks out an element in eachM-orbit so thatM has the same isotropy subgroup at each of those elements. In otherwords in eachM-orbit onu∗ we measurably choose an elementλ = σ(Ad∗(M)λ )such that those isotropy subgroupsMλ are all the same. Let us denote

M♦: isotropy subgroup ofM at σ(Ad∗(M)λ ) for everyλ ∈ u∗ (22)

We replaceMλ by M♦, independent ofλ ∈ u∗, in Proposition 7.2. That lets usassemble to representations of Proposition 7.2 for a Plancherel Formula, as follows.SinceM is compact, we have the Schwartz spaceC (NM) just as in the discussionof C (N) between (6) and Theorem 3.5, except that the pullback exp∗C (NM) 6=C (n+m). The same applies toC (NA) andC (NAM)

Page 12: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

12 Joseph A. Wolf

Proposition 7.3.Let f ∈ C (NM) and write( fm)(n) = f (nm) = (n f )(m) for n∈ Nand m∈ M. The Plancherel density atIndNM

NM♦(π†

λ ⊗ γ) is (dimγ)|Pf(bλ )| and thePlancherel Formula for NM is

f (nm) = c∫

u∗/Ad∗(M)∑

F (λ )traceηλ ,γ(n fm) ·dim(γ) · |Pf(bλ )|dλ

where c= 2d1+···+dmd1!d2! . . .dm! , from (6), as inTheorem 3.5.

8. Minimal Parabolics: MAN and AN.

Let G be a separable locally compact group of type I. Then [14,§1] the Plancherelformula forG has form

f (x) =∫

Gtraceπ(D(r(x) f ))dµG(π) (23)

whereD is an invertible positive self adjoint operator onL2(G), conjugation–semi-invariant of weight equal to the modular functionδG, andµ is a positive Borel mea-sure on the unitary dualG. If G is unimodular thenD is the identity and (23) reducesto the usual Plancherel formula. The point is that semi-invariance ofD compensatesany lack of unimodularity. See [14,§1] for a detailed discussion.D⊗µ is unique (upto normalization of Haar measures) and one tries to find a “best” choice ofD. Givenany such pair(D,µ) we refer toD as aDixmier–Pukanszky OperatoronG and toµas the associatedPlancherel measureonG. We will construct a Dixmier–PukanszkyOperator from the Pfaffian polynomial Pf(bλ ).

Let δAN andδQ denote the modular functions onAN and onQ= MAN. As M iscompact and AdQ(N) is unipotent onp, they are determined by their restrictions toA, where they are given byδ (exp(ξ )) = exp(trace(ad(ξ ))) with ξ = loga∈ a.

Lemma 8.1.Let ξ ∈ a. Then12(dimlr +dimzr) ∈ Z for 1≦ r ≦ m and

(i) the trace ofad(ξ ) on lr is 12(dimlr +dimzr)βr(ξ ),

(ii) the trace ofad(ξ ) onn and onp is 12 ∑r(dimlr +dimzr)βr(ξ ),

(iii) the determinant ofAd(exp(ξ )) onn and onp is ∏r exp(βr(ξ ))12 (dimlr+dimzr ),

(iv) δQ(man) = ∏r exp(βr(loga))12 (dimlr+dimzr ) andδAN = δQ|AN.

Now compute

Lemma 8.2.Letξ ∈ a and a= exp(ξ )∈A. Thenad(ξ )Pf =(1

2 ∑r dim(lr/zr)βr(ξ ))

Pf

andAd(a)Pf =(

∏r exp(βr(ξ ))12 dim(lr/dimzr )

)Pf.

At this point it is convenient to introduce some notation anddefinitions.

Definition 8.3. The algebras is the quasi–centerof n. The polynomial functionDets∗(λ ) := ∏r(βr(λ ))dimgβr on s∗ is thequasi–center determinant.

Page 13: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 13

Forξ ∈ a anda= exp(ξ )∈A compute(Ad(a)Dets∗)(λ ) =Dets∗(Ad∗(a−1)(λ ))= ∏r(βr(Ad(a−1)∗λ ))dimgβr = ∏r(βr(exp(βr(ξ ))λ ))dimgβr . In other words,

Lemma 8.4.Let a= exp(ξ ) ∈ A. ThenAd(a)Dets∗ =(∏r exp(βr(ξ ))dimzr

)Dets∗

whereξ = loga∈ a.

Combining Lemmas 8.1, 8.2 and 8.4 we have

Proposition 8.5.The productPf ·Dets∗ is anAd(MAN)–semi–invariant (and thusAd(AN)–semi–invariant) polynomial ons∗ of degree1

2(dimn+dims) and of weightequal to the respective modular functions of Q and AN.

Fromn= v+ s we haveN =VSwhereV = exp(v) andS= exp(s). Now define

D : Fourier transform of Pf·Dets∗ , acting on theSvariable ofN =VS. (24)

Theorem 8.6.The operator D of(24) is an invertible self–adjoint differential op-erator of degree1

2(dimn+dims) on L2(MAN) with dense domainC (MAN), andit is Ad(MAN)-semi-invariant of weight equal to the modular functionδMAN . Inother words|D| is a Dixmier–Pukanszky Operator on MAN with domain equal tothe space of rapidly decreasing C∞ functions. This applies as well to AN.

Sinceλ ∈ t∗ has nonzero projection on each summandz∗r of s∗, anda∈ A actsby the positive real scalar exp(βr(log(a))) on zr ,

Aλ = exp({ξ ∈ a | eachβr(ξ ) = 0}), independent ofλ ∈ t∗. (25)

Because of this independence, and usinga♦ = {ξ ∈ a | eachβr(ξ ) = 0}, we define

A♦ = Aλ for any (and thus for all)λ ∈ t∗. (26)

Lemma 8.7.If λ ∈ σ(u∗) then the stabilizer(MA)λ = M♦A♦ .

There is no problem with the Mackey obstruction:

Lemma 8.8.Let λ ∈ σ(u∗). Recall the extension(before (21))π†λ of πλ to NM♦ .

Thenπ†λ extends toπλ ∈ NM♦A♦ with the same representation space asπλ .

Whenλ ∈ σ(u∗), A♦ consists of the unitary characters exp(iφ) : a 7→ eiφ(loga)

with φ ∈ a∗♦. The representations ofQ corresponding toλ are the

πλ ,γ,φ := IndNMANM♦A♦

(πλ ⊗ γ ⊗exp(iφ)) whereγ ∈ M♦ andφ ∈ a∗♦ . (27)

Ad∗(A) fixesγ becauseA centralizesM, and it fixesφ becauseA is commutative, so

πλ ,γ,φ ·Ad((ma)−1) = πAd∗(ma)λ ,γ,φ (28)

Page 14: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

14 Joseph A. Wolf

Proposition 8.9.Plancherel measure for Q is concentrated on the the set of allπλ ,γ,φ for λ ∈ σ(u∗), γ ∈ M♦ andφ ∈ a∗♦ . The equivalence class ofπλ ,γ,φ dependsonly on(Ad∗(MA)λ ,γ,φ).

Representations ofAN are the caseγ = 1. In effect, letπ ′λ denote the obvious

extensionπλ |AN of the stepwise square integrable representationπλ from N to NA♦

whereπλ is given by Lemma 8.8. Denote

πλ ,φ = IndNANA♦

(π ′λ ⊗exp(iφ)) whereλ ∈ u∗ andφ ∈ a∗♦. (29)

Corollary 8.10. Plancherel measure for AN is concentrated on the set of allπλ ,φ forλ ∈ u∗ andφ ∈ a∗♦ . The equivalence class ofπλ ,φ depends only on(Ad∗(MA)λ ,φ).

A result of C.C. Moore implies

Lemma 8.11.ThePf–nonsingular principal orbit setu∗ is a finite union of openAd∗(MA)–orbits.

Let{O1 , . . .Ov} denote the (open) Ad∗(MA)–orbits onu∗. Denoteλi = σ(Oi), soOi = Ad∗(MA)λi and(MA)λi

= M♦A♦ for 1≦ i ≦ v. Then Proposition 8.9 becomes

Theorem 8.12.Plancherel measure for MAN is concentrated on the set(of equiva-lence classes of) unitary representationsπλi ,γ,φ for 1≦ i ≦ v, γ ∈ M♦ andφ ∈ a∗♦ .

Now the Plancherel Theorem forQ= MAN isThe Plancherel Formula (or Fourier Inversion Formula) forMAN is

Theorem 8.13.Let Q= MAN be a minimal parabolic subgroup of the real re-ductive Lie group G. Givenπλ ,γ,φ ∈ MAN as described in(27) let Θπλ ,γ,φ : h 7→

traceπλ ,γ,φ (h) denote its distribution character. ThenΘπλ ,γ,φ is a tempered distribu-tion. If f ∈ C (MAN) then

f (x) = cv

∑i=1

∑γ∈M♦

a∗♦

Θπλi ,γ,φ(D(r(x) f ))|Pf(bλi

)|dimγ dφ

where c> 0 depends on normalizations of Haar measures.

The Plancherel Theorem forNA follows similar lines. For the main computationin the proof of Theorem 8.13 we omitM andγ. That gives

a∗♦

traceπλ0,φ (Dh)dφ =

Ad∗(A)λ0

traceπλ (h)|Pf(bλ )|dλ (30)

In order to go from an Ad∗(A)λ0 to an integral overu∗ we useM to parameterize thespace of Ad∗(A)–orbits onu∗. If λ ∈ u∗ one proves Ad∗(A)λ ∩Ad∗(M)λ = {λ}.That leads to

Page 15: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 15

Proposition 8.14.Plancherel measure for NA is concentrated on the equivalenceclasses of representationsπλ ,φ = IndNA

NA♦(π ′

λ ⊗exp(iφ)) whereλ ∈Si :=Ad∗(M)λi ,

1 ≦ i ≦ v, π ′λ extendsπλ from N to NA⋄ and φ ∈ a∗⋄ . Representationsπλ ,φ and

πλ ′,φ ′ are equivalent if and only ifλ ′ ∈ Ad∗(A)λ and φ ′ = φ . Further, πλ ,φ |N =∫a∈A/A♦

πAd∗(a)λ da.

Theorem 8.15.Let Q= MAN be a minimal parabolic subgroup of the real reduc-tive Lie group G. Ifπλ ,φ ∈ AN letΘπλ ,φ : h 7→ traceπλ ,φ (h) denote its distributioncharacter. ThenΘπλ ,φ is a tempered distribution. If f∈ C (AN) then

f (x) = cv

∑i=1

λ∈Ad∗(M)λi

a∗♦

traceπλ ,φ (D(r(x) f ))|Pf(bλ )|dλdφ .

where c> 0 depends on normalizations of Haar measures.

9. Parabolic Subgroups in General: the Nilradical.

In Sections 7 and 8 we studied minimal parabolic subgroupsQ = MAN in simpleLie groups, along with certain of their subgroupsMN andAN. This section and thenext form a glance at more general parabolics. This materialis taken from [32],which is a work in progress, and is limited to the part that I’ve written down. Westart with the structure of the nilradical.

The condition (c) of (5) does not always hold for nilradicalsof parabolic sub-groups. In this section and the next we weaken (5) to

N =L1L2 . . .Lm−1Lm where

(a) eachLr has unitary representations with coefficients inL2(Lr/Zr),

(b) eachNr := L1L2 . . .Lr = Nr−1⋊Lr semidirect,

(c) if r ≧ s then[lr ,zs] = 0.

(31)

The conditions of (31) are sufficient to construct stepwise square integrable repre-sentations, but are not always sufficient to compute the Pfaffian that is the Planchereldensity. So we refer to (5) as thestrong computability conditionand make make useof theweak computability condition

Let lr = l′r ⊕ l′′r wherel′′r ⊂ zr andvr ⊂ l′r ; then[lr , ls]⊂ l′′s +vs for r > s. (32)

where we retainlr = zr +vr andn= s+v.

Consider an arbitrary parabolic subgroup ofG. It contains a minimal parabolicQ = MAN. Let Ψ denote the set of simple roots for the positive system∆+(g,a).Then the parabolic subgroups ofG that containQ are in one to one correspondencewith the subsetsΦ ⊂Ψ , sayQΦ ↔ Φ, as follows. DenoteΨ = {ψi} and set

Page 16: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

16 Joseph A. Wolf

Φ red ={

α = ∑ψi∈Ψ niψi ∈ ∆(g,a) | ni = 0 wheneverψi /∈ Φ}

Φnil ={

α = ∑ψi∈Ψ niψi ∈ ∆+(g,a) | ni > 0 for someψi /∈ Φ}.

(33)

On the Lie algebra level,qΦ =mΦ + aΦ +nΦ where

aΦ = {ξ ∈ a | ψ(ξ ) = 0 for all ψ ∈ Φ} = Φ⊥ ,

mΦ + aΦ is the centralizer ofaΦ in g, somΦ has root systemΦ red, and

nΦ = ∑α∈Φnil gα , nilradical ofqΦ , sum of the positiveaΦ–root spaces.

(34)

Sincen= ∑r lr , as given in (17) and 18) we have

nΦ = ∑r(nΦ ∩ lr) = ∑r

((gβr ∩nΦ)+∑∆+

r(gα ∩nΦ)

). (35)

As ad(m) is irreducible on each restricted root space, ifα ∈ {βr}∪∆+r thengα ∩nΦ

is 0 or all ofgα .

Lemma 9.1.Supposegβr ∩nΦ = 0. Thenlr ∩nΦ = 0.

Lemma 9.2.Supposegβr ∩nΦ 6= 0. Define Jr ⊂ ∆+r by lr ∩nΦ = gβr +∑Jr gα . De-

compose Jr = J′r ∪J′′r where J′r = {α ∈ Jr | σr α ∈ Jr} and J′′r = {α ∈ Jr | σrα /∈ Jr}.Thengβr +∑J′′r gα belongs to a singleaΦ–root space innΦ , i.e.α|aΦ = βr |aΦ , foreveryα ∈ J′′r .

Lemma 9.3.Supposelr ∩ nΦ 6= 0. Then the algebralr ∩ nΦ has centergβr +

∑J′′r gα , and lr ∩ nΦ = (gβr +∑J′′r gα) + (∑J′r gα)). Further, lr ∩ nΦ =(∑J′′r gα

)⊕(

gβr +(∑J′r gα

))direct sum of ideals.

It will be convenient to define sets of simpleaΦ–roots

Ψ1 =Ψ andΨs+1 = {ψ ∈Ψ | 〈ψ ,βi〉= 0 for 1≦ i ≦ s}. (36)

Note thatΨr is the simple root system for{α ∈ ∆+(g,a) | α ⊥ βi for i < r}.

Lemma 9.4.If r > s then[lr ∩nΦ , gβs +∑J′′s gα ] = 0.

For our dealings with arbitrary parabolics it is not sufficient to consider linearfunctionals on∑r gβr . Instead we have to look at linear functionals on∑r

(gβr +

∑J′′r gα). of the formλ = ∑λr whereλr ∈ g∗βr

such thatbλr is nondegenerate on∑r ∑J′r gα . We know that (5(c)) holds for the nilradical of the minimal parabolicqthat containsqΦ . In view of Lemma 9.4 it follows thatbλ (lr , ls) = λ ([lr , ls] = 0 forr > s. For this particular type ofλ , the bilinear formbλ has kernel∑r

(gβs+∑J′′s gα

)

and is nondegenerate on∑r ∑J′r gα . ThenNΦ = (L1∩NΦ )(L2 ∩NΦ ) . . . (Lm∩NΦ )satisfies the first two conditions of (5). That is enough to carry out the constructionof stepwise square integrable representationsπλ of NΦ , but one needs to do moreto deal with Pfaffian polynomials as in (5(c)) and (32).

Page 17: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 17

Let I1 = {i | βi |aΦ = βq1|aΦ } whereq1 is the first index of (5) withβq1|aΦ 6= 0.Next,I2 = {i | βi |aΦ = βq2|aΦ } whereq2 is the first index of (5) such thatq2 /∈ I1 andβq2|aΦ 6= 0. Continuing as long as possible,Ik = {i | βi |aΦ = βqk|aΦ } whereqk is thefirst index of (5) such thatqk /∈ (I1 ∪ ·· · ∪ Ik−1) andβqk|aΦ 6= 0. ThenI1 ∪ ·· · ∪ Iℓconsists of all the indicesi for which βi|aΦ 6= 0. For 1≦ j ≦ ℓ define

lΦ , j = ∑i∈I j(li ∩nΦ) =

(∑i∈I j

li

)∩nΦ andl†Φ , j = ∑k≧ j

lΦ ,k . (37)

Lemma 9.5.If k ≧ j then [lΦ ,k, lΦ , j ] ⊂ lΦ , j . For each index j,lΦ , j and l†Φ , j are

subalgebras ofnΦ andlΦ , j is an ideal inl†Φ , j .

Lemma 9.6.If k > j then[lΦ ,k , lΦ , j ]∩∑i∈I jgβi

= 0.

In the notation of Lemma 9.2, ifr ∈ I j then

lr ∩nΦ = l′r + l′′r wherel′r = gβr +∑J′rgα andl′′r = ∑J′′r

gα . (38)

For 1≦ j ≦ ℓ definezΦ , j = ∑i∈I j

(gβi+ l′′i ) (39)

and decompose

lΦ , j = l′Φ , j + l′′Φ , j wherel′Φ , j = ∑i∈I jl′i andl′′Φ , j = ∑i∈I j

l′′i . (40)

Lemma 9.7.Recall l†Φ , j = ∑k≧ j lΦ ,k from (37). For each j, bothzΦ , j and l′′Φ , j are

central ideals inl†Φ , j , andzΦ , j is the center oflΦ , j .

Decompose

nΦ = zΦ +vΦ wherezΦ = ∑jzΦ , j , vΦ = ∑

jvΦ , j andvΦ , j = ∑

i∈I j

∑α∈J′i

gα . (41)

Then Lemma 9.7 gives us (32) for thelΦ , j : lΦ , j = l′Φ , j ⊕ l′′Φ , j with l′′Φ , j ⊂ zΦ , j andvΦ , j ⊂ l′Φ , j .

Lemma 9.8.For genericλ j ∈ z∗Φ , j the kernel of bλ jon lΦ , j is just zΦ , j , in other

words bλ jis is nondegenerate onvΦ , j ≃ lΦ , j/zΦ , j . In particular LΦ , j has square

integrable representations.

Theorem 9.9.Let G be a real reductive Lie group and Q a real parabolic subgroup.Express Q= QΦ in the notation of(33) and (34). Then its nilradical NΦ has de-composition NΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ that satisfies the conditions of(5) and (32) asfollows. The center ZΦ , j of LΦ , j is the analytic subgroup forzΦ , j and

Page 18: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

18 Joseph A. Wolf

(a)each LΦ , j has unitary representations with coefficients in L2(LΦ , j/ZΦ , j),

(b) each NΦ , j := LΦ ,1LΦ ,2 . . .LΦ , j is a normal subgroup of NΦwith NΦ , j = NΦ , j−1⋊LΦ , j semidirect,

(c) [lΦ ,k,zΦ , j ] = 0 and[lΦ ,k, lΦ , j ]⊂ vΦ , j + l′′Φ , j for k> j.

(42)

In particular NΦ has stepwise square integrable representations relative to the de-composition NΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ .

10. Amenable Subgroups of Semisimple Lie Groups.

In this section we apply the results of Section 9 to certain important subgroups ofthe parabolicQΦ = MΦ AΦNΦ , specifically its amenable subgroupsAΦNΦ , UΦNΦandUΦAΦNΦ whereUΦ is a maximal compact subgroup ofMΦ .

The theory of the groupUΦNΦ goes exactly as in Section 7. WhenNΦ =LΦ ,1LΦ ,2 . . .LΦ ,ℓ is weakly invariant we can proceed more or less as in [29]. Theargument, but not the final result, will make use of

Definition 10.1.The decompositionNΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ of Theorem 9.9 isin-variant if each ad(mΦ )zΦ , j = zΦ , j , equivalently if each Ad(MΦ )zΦ , j = zΦ , j , inother words wheneverzΦ , j = g[Φ ,β j0 ]

. The decompositionNΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ

is weakly invariantif each Ad(UΦ)zΦ , j = zΦ , j . ♦

Set

r∗Φ = {λ ∈ s∗Φ | P(λ ) 6= 0 and Ad(UΦ)λ is a principalUΦ –orbit ons∗Φ}. (43)

Thenr∗Φ is dense, open andUΦ–invariant ins∗Φ . By definition of principal orbit theisotropy subgroups ofUΦ at the various points ofr∗Φ are conjugate, and we take ameasurable sectionσ to r∗Φ → UΦ\r∗Φ on whose image all the isotropy subgroupsare the same,

U ′Φ : isotropy subgroup ofUΦ at σ(UΦ(λ )), independent ofλ ∈ r∗Φ . (44)

The principal isotropy subgroupsU ′Φ are pinned down in [11]. Givenλ ∈ r∗Φ and

γ ∈ U ′Φ let π†

λ denote the extension ofπλ to a representation ofU ′ΦNΦ on the space

of πλ and defineπλ ,γ = IndUΦ NΦ

U ′Φ NΦ

(γ ⊗π†λ ). (45)

The first result in this setting, as in [29, Proposition 3.3],is

Theorem 10.2.Suppose that NΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ as in(31). Then the Plancherel

density onUΦNΦ is concentrated on the representationsπλ ,γ of (45), the Planchereldensity atπλ ,γ is (dimγ)|P(λ )|, and the Plancherel Formula for UΦNΦ is

Page 19: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 19

f (un) = c∫

r∗Φ/Ad∗(UΦ )∑γ∈U ′

ΦtraceIndUΦ NΦ

U ′Φ NΦ

run( f ) ·dim(γ) · |P(λ )|dλ

where c= 2d1+···+dℓd1!d2! . . .dℓ! as in(6).

Recall the notion of amenability.. Ameanon a locally compact groupH is alinear functionalµ on L∞(H) of norm 1 and such thatµ( f ) ≧ 0 for all real–valuedf ≧ 0. H is amenableif it has a left–invariant mean. Solvable groups and compactgroups are amenable, as are extensions of amenable groups byamenable subgroups.In particularEΦ :=UΦ AΦNΦ and its closed subgroups are amenable.

We need a technical condition [15, p. 132]. LetH be the group of real points ina linear algebraic group whose rational points are Zariski dense, letA be a maximalR–split torus inH, let ZH(A) denote the centralizer ofA in H, and letH0 be thealgebraic connected component of the identity inH. ThenH is isotropically con-nectedif H = H0 ·ZH(A). More generally we will say that a subgroupH ⊂ G isisotropically connectedif the algebraic hull of AdG(H) is isotropically connected.

Proposition 10.2.[15, Theorem 3.2].The groups EΦ := UΦAΦNΦ are maximalamenable subgroups of G. They are isotropically connected and self–normalizing.The variousΦ ⊂Ψ are mutually non–conjugate. An amenable subgroup H⊂ G iscontained in some EΦ if and only if it is isotropically connected.

The isotropy subgroups are the same at everyλ ∈ t∗Φ ,

A′Φ : isotropy subgroup ofAΦ at λ ∈ r∗Φ . (46)

Given a stepwise square integrable representationπλ whereλ ∈ s∗Φ , write π†λ for

the extension ofπλ to a representation ofA′ΦNΦ on the same Hilbert space. That

extension exists because the Mackey obstruction vanishes.The representations ofA′

ΦNΦ corresponding toπλ are the

πλ ,φ := IndAΦ NΦA′

Φ NΦ(exp(iφ)⊗π†

λ ) whereφ ∈ a′Φ . (47)

Note also that

πλ ,φ ·Ad(an) = πAd∗(a)λ ,φ for a∈ AΦ andn∈ NΦ . (48)

The resulting formulaf (x) =∫

H traceπ(D(r(x) f ))dµH(π), H = AΦNΦ , is

Theorem 10.3.Let QΦ = MΦAΦ NΦ be a parabolic subgroup of the real reductiveLie group G. Givenπλ ,φ ∈ AΦNΦ as described in(47), its distribution characterΘπλ ,φ : h 7→ traceπλ ,φ (h) is a tempered distribution. If f∈ C (AΦNΦ ) then

f (x) = c∫

(a′Φ )∗

(∫

s∗Φ/Ad∗(AΦ )Θπλ ,φ (D(r(x) f ))|Pf(bλ )|dλ

)dφ

where c= 2d1+···+dℓd1!d2! . . .dℓ! .

Page 20: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

20 Joseph A. Wolf

The representations ofUΦAΦNΦ corresponding toπλ are the

πλ ,φ ,γ := IndUΦ AΦ NΦU ′

Φ A′Φ NΦ

(γ ⊗exp(iφ)⊗π†λ ) whereφ ∈ a′Φ andγ ∈ U ′

Φ . (49)

Combining Theorems 10.2 and 10.3 we arrive at

Theorem 10.4.Let QΦ = MΦAΦ NΦ be a parabolic subgroup of the real reductiveLie group G and decompose NΦ = LΦ ,1LΦ ,2 . . .LΦ ,ℓ as in(31). Then the Plancherel

density on UΦAΦNΦ is concentrated on theπλ ,φ ,γ of (49), the Plancherel densityat πλ ,φ ,γ is (dimγ)|P(λ )|, the distribution characterΘπλ ,φ ,γ : h 7→ traceπλ ,φ ,γ(h) istempered, and if f∈ C (UΦAΦNΦ ) then

f (x) = c∑U ′

Φ

(a′Φ )∗

(∫

s∗Φ/Ad∗(UΦ AΦ )Θπλ ,φ ,γ (D(r(x) f ))deg(γ) |Pf(bλ )|dλ

)dφ

where c= 2d1+···+dℓd1!d2! . . .dℓ! .

References

1. L. Auslander et al, “Flows on Homogeneous Spaces”, Ann. Math. Studies53, 1963.2. I. Beltita & D. Beltita, Coadjoint orbits of stepwise square integrable representations,

to appear.{arXiv:1408.1857}3. I. Beltita & J. Ludwig, Spectral synthesis for coadjoint orbits of nilpotent Lie groups,

to appear.{arXiv:1412.6323}4. I. Beltita & D. Beltita, Representations of nilpotent Liegroups via measurable dy-

namical systems{arXiv:1510.05272}5. G. Bredon, Introduction to Compact Transformation Groups, Academic Press, 1972.6. W. Casselman, Introduction to the Schwartz space ofΓ \G, Canadian J. Math.40

(1989), 285–320.7. M. Duflo, Sur les extensions des representations irreductibles des groups de Lie

nilpotents, Ann. Sci. de l’Ecole Norm. Super., 4ieme serie5 (1972), 71–120.8. I. Dimitrov, I. Penkov & J. A. Wolf, A Bott–Borel–Weil theory for direct limits of

algebraic groups, Amer. J of Math.124(2002), 955–998.9. J. Faraut, Infinite dimensional harmonic analysis and probability, in “Probability

Measures on Groups: Recent Directions and Trends,” ed. S. G.Dani & P. Graczyk,Narosa, New Delhi, 2006.

10. V. V. Gorbatsevich, A. L. Onishchik & E. B. Vinberg, Foundations of Lie Theory andLie Transformation Groups, Springer, 1997.

11. W.-C. Hsiang & W.-Y. Hsiang, Differentiable actions of compact connected classicalgroups II, Annals of Math.92 (1970), 189–223.

12. A. A. Kirillov, Unitary representations of nilpotent Lie groups, Uspekhi Math. Nauk17 (1962), 57–110 (English: Russian Math. Surveys17 (1962), 53–104).

13. M. Kramer, Spharische Untergruppen in kompakten zusammenhangenden Liegrup-pen, Compositio Math.38 (1979), 129–153.

14. R. L. Lipsman & J. A. Wolf, The Plancherel formula for parabolic subgroups of theclassical groups, Journal D’Analyse Mathematique,34 (1978), 120–161.

15. C. C. Moore, Amenable subgroups of semi-simple Lie groups and proximal flows.Israel J. Math.34 (1979), 121–138.

Page 21: arXiv:1511.09064v1 [math.RT] 29 Nov 2015matrix coefficients for unitary representations of nilpote nt Lie groups. These have several consequences for the geometry of weakly symmetric

Stepwise Square Integrable Representations 21

16. C. C. Moore, Decomposition of unitary representations defined by discrete subgroupsof nilpotent groups, Ann. Math.82 (1965), 146–182

17. C. C. Moore & J. A. Wolf, Square integrable representations of nilpotent groups.Transactions of the American Mathematical Society,185(1973), 445–462.

18. S. de Neymet Urbina (con la colaboracion de R. Jimenez Benıtez), Introduccion a losGrupos Topologicos de Transformaciones, Sociedad Matem´atica Mexicana, 2005.

19. G. I. Ol’shanskii, Unitary representations of infinite dimensional pairs(G,K) and theformalism of R. Howe, in “Representations of Lie Groups and Related Topics, ed.A. M. Vershik & D. P. Zhelobenko,” Advanced Studies Contemp.Math. 7, Gordon& Breach, 1990.

20. L. Pukanszky, On characters and the Plancherel formulaof nilpotent groups, J. Func-tional Analysis1 (1967), 255–280.

21. M. S. Raghunathan, “Discrete Subgroups of Lie Groups”, Ergebnisse der Mathe-matik und ihrer Grenzgebeite68, 1972.

22. E. B. Vinberg, Commutative homogeneous spaces and co–isotropic symplectic ac-tions, Russian Math. Surveys56 (2001), 1–60.

23. E. B. Vinberg, Commutative homogeneous spaces of Heisenberg type, TransMoscow Math. Soc.64 (2003), 45–78.

24. J. A. Wolf, Classification and Fourier inversion for parabolic subgroups with squareintegrable nilradical. Memoirs of the American Mathematical Society, Number 225,1979.

25. J. A. Wolf, Direct limits of principal series representations, Compositio Mathematica,141(2005), 1504–1530.

26. J. A. Wolf, Harmonic Analysis on Commutative Spaces, Math. Surveys & Mono-graphs vol. 142, Amer. Math. Soc., 2007.

27. J. A. Wolf, Infinite dimensional multiplicity free spaces II: Limits of commutativenilmanifolds, to appear.

28. J. A. Wolf, Stepwise square integrable representationsof nilpotent Lie groups, Math-ematische Annalen vol. 357 (2013), pp. 895–914.{arXiv: see 1212.1908}

29. J. A. Wolf, The Plancherel Formula for Minimal ParabolicSubgroups, Journal of LieTheory, vol. 24 (2014), pp. 791–808.{arXiv: 1306.6392 (math RT)}

30. J. A. Wolf, Stepwise square integrable representationsfor locally nilpotent Liegroups, Transformation Groups, vol. 20 (2015), pp. 863–879. {arXiv:1402.3828(math RT, math FA)}

31. J. A. Wolf, On the analytic structure of commutative nilmanifolds, The Journal ofGeometric Analysis, to appear.{arXiv:1407.0399 (math RT, math DG)}

32. J. A. Wolf, Stepwise square integrability for nilradicals of parabolic subgroups andmaximal amenable subgroups, work in progress.

33. O. S. Yakimova, Weakly symmetric riemannian manifolds with reductive isometrygroup, Math. USSR Sbornik195(2004), 599–614.

34. O. S. Yakimova, “Gelfand Pairs,” Bonner Math. Schriften(Universitat Bonn)374,2005.

35. O. S. Yakimova, Principal Gelfand pairs, Transformation Groups11 (2006), 305–335.