arX

iv:q

uant

-ph/

0607

204v

1 2

8 Ju

l 200

6

Permutation groups, minimal degrees and quantum computing

Julia KempeCNRS & LRI, Universite Paris-Sud

91405 Orsay Cedex, France

Laszlo PyberMathematical Institute of the Hungarian Academy of Sciences

P.O. Box 127, Budapest, Hungary H-1364

Aner ShalevInstitute of MathematicsThe Hebrew UniversityJerusalem 91904, Israel

October 17, 2018

Abstract

We study permutation groups of given minimal degree withoutthe classical primitivity as-sumption. We provide sharp upper bounds on the order of a permutation groupH ≤ Sn ofminimal degreem and on the number of its elements of any given support. These results con-tribute to the foundations of a non-commutative coding theory.

A main application of our results concerns the Hidden Subgroup Problem forSn in Quan-tum Computing. We completely characterize the hidden subgroups ofSn that can be distin-guished from identity with weak Quantum Fourier Sampling, showing these are exactly thesubgroups with bounded minimal degree. This implies that the weak standard method forSn

has no advantage whatsoever over classical exhaustive search.

1

1 Introduction

Let Sn denote the symmetric group on1, . . . , n. For a permutationh ∈ Sn define its supportsupp(h) by

supp(h) = i ∈ 1, . . . , n : h(i) 6= i.The minimal degreem(H) of a permutation group1 6= H ≤ Sn is defined to be the minimalnumber of points moved by a non-identity element ofH. In other words,

m(H) = min|supp(h)| : 1 6= h ∈ H.

This notion goes back to the 19th century, and plays an important role in the theory of finite permu-tation groups since the days of Jordan [Jor73, Jor75]. Particular attention was given to the minimaldegree ofprimitive permutation groups. Recall that a permutation group is called primitive if it istransitive and doesn’t preserve a non-trivial block system. LetH < Sn be a primitive permutationgroup not containingAn. Jordan proved thatm(H) goes to infinity asn goes to infinity. Babai[Bab81] showed that under the above conditions we actually have that

m(H) ≥√n− 1

2.

This result is essentially best possible. However, if we exclude certain primitive groups and usethe Classification of Finite Simple Groups (CFSG), sharper bounds can be obtained. Indeed, it wasshown by Liebeck and Saxl in [LS91] thatm(H) ≥ n/3 with a given list of exceptions. This lowerbound was improved by Guralnick and Magaard in [GM98] ton/2 (with prescribed exceptions).See also Cameron [Cam81] for the impact of the Classificationon the theory of finite permutationgroups and primitive groups in particular.

In spite of considerable progress in the study of the minimaldegree of primitive groups, muchless is known in the non-primitive case. One of the purposes of this paper is to study permutationgroups of given minimal degree without assuming primitivity or even transitivity.

A basic question in this field is: how large can a permutation groupH of degreen and minimaldegreem be? An easy classical upper bound is|H| ≤ nn−m+1. Indeed, this follows from the factthat a permutationh ∈ H is uniquely determined by its action on1, . . . , n−m+ 1.

Better bounds were given by Liebeck [L82, L84] under the assumption thatH is transitive. Ourfirst result extends Liebeck’s theorem to arbitrary permutation groups.

Theorem A. LetH ≤ Sn be a permutation group with minimal degreem = m(H).1) If m ≤ log2 n, then|H| ≤ n10n/m.2) If m ≥ log2 n, then|H| ≤ 210n.

Theorem A is essentially best possible. For example, consider the groupH = S2n/m < Sn

acting on2n/m blocks of sizem/2. Then the minimal degree ofH is m and |H| = (2n/m)!which is of the formn(2−o(1))n/m whenm ≤ log2 n. Up to a constant in the exponent, this showsthat part (1) of Theorem A is tight.

Note also that ifH ≤ Sn is transitive of minimal degreem and base sizeb, thenbm ≥ n (seee.g. [DM96], p. 80), and this implies|H| ≥ 2b ≥ 2n/m.

Subgroups ofSn of given minimal degreem can be regarded as non-commutative analogues oflinear codes with minimal distancem. Recall that in coding theory [MS77] a fundamental question

2

is: how large can a subspace ofGF (q)n with minimal distancem be? Replacing the Abelian groupGF (q)n by the symmetric groupSn we may ask a similar question in this context. Theorem Aprovides a rather sharp answer.

Note that any binary linear code insideGF (2)n/2 can be embedded naturally as a subgroup

of Sn/22 < Sn. Thus classical coding theory provides a rich source of constructions of permuta-

tion groups of large minimal degree. In particular the (obvious) Gilbert-Varshamov lower bound([GGL96] p. 781, remark after Thm. 3.5) applied to linear codes produces exponentially large ele-mentary Abelian permutation groups with large minimal degree, e.g.m > n/8. This demonstratesthe tightness of part (2) of Theorem A, even whenm is very large.

Another classical question in coding theory is the study of theweight distribution, namely count-ing elements of weightk in a code with minimal distancem. The analogous question for permu-tation groups is counting the number of elements of supportk in a permutation group of minimaldegreem. Given a permutation groupH ≤ Sn define

Hk = h ∈ H : |supp(h)| = k,

the subset of elements of supportk in H. In our second result, which is the most technicallydemanding, we bound the size ofHk.

Theorem B. There exists absolute constantsb, ε > 0 such that if a subgroupH ≤ Sn has minimaldegreem ≥ b then

|Hk| ≤ n−εm

(n

k

) 12

(k!)14 .

The theorem has an interesting consequence for the number ofelements of minimal support. Ifk = m ≤ n2ε then(k!)1/4 ≤ nεm/2 and this implies

|Hm| ≤ n−εm/2

(n

m

)1/2

.

This upper bound is essentially tight. To show this we use some results from coding theory andthe above embedding of binary codes inSn. Consider the well known Goppa code [G70] and theestimates for the number of code words of minimal weight [LL97]. For a binary Goppa code overGF (2)n/2, in the regime of smallt (t ≪ √

log n), the number of code words of minimum weight2t+ 1 is roughly (up to a constant factor)

(n/2

2t+ 1

)(n

2)−t.

Embedding this code intoSn as above, we obtain a subgroupH < Sn of minimal degreem = 4t+2satisfying

|Hm| ≥ cn−m/4

(n

m

)12

for some constantc > 0. This demonstrates the tightness of Theorem B in the regime of smallm.

A main motivation behind Theorem B, besides the study of weight distributions of non-commutativecodes, comes from Quantum Computing. A central problem in Quantum Computing is the Hidden

3

Subgroup Problem (HSP), which we state below. LetG be a finite group andH ≤ G a subgroup.Given a functionf : G → S that is constant on (left)-cosetsgH of H and takes different valuesfor different cosets, determine a set of generators forH. The decision version of this problem is todetermine whether there is a non-identity hidden subgroup or not.

Note that giveng ∈ G we haveg ∈ H if and only if f(g) = f(1). Using classical searchwe may therefore perform membership tests, and once we find a non-identity elementg ∈ H wemay conclude thatH 6= 1. However, the aim is to decide whether or notH = 1 in polynomialtime, namely after(log |G|)c steps. Complete enumeration over the elementsg ∈ G is thereforenot efficient. The question is whether a quantum computer cansolve the HSP efficiently (giving thecorrect answer in polynomial time with a very high probability).

The Hidden Subgroup Problem plays a central role in Quantum Computing. Nearly all quantumalgorithms which significantly improve the known classicalalgorithms, like factoring and discretelog, solve the Abelian version of this problem by the so called standard method of Quantum FourierSampling. One of the most important questions is whether thestandard method can efficiently solvethe non-AbelianHSP, especially for the symmetric groupG = Sn. This latter case in particularwould yield a quantum algorithm for the Graph Isomorphism Problem, for which no efficient clas-sical algorithm is known. For more details on Quantum Computing, the HSP, and the standardmethod see Section 2.

To state our main quantum-theoretic application in a precise mathematical way we need somenotation. Given a finite groupG let Irr(G) denote the set of (complex) irreducible representationsof G (up to equivalence). Forρ ∈ IrrG let dρ denote its dimension andχρ its character.

Given a subgroupH ≤ G, define

DH =1

|G|∑

ρ∈IrrGdρ|

∑

h∈H,h 6=1

χρ(h)|. (1)

Roughly speaking,DH measures theL1-distance between a (non-commutative) Fourier trans-form of the characteristic function ofH and that of the characteristic function of the identity.

We say that a subgroupH ≤ G is distinguishableif

DH ≥ (log |G|)−c

for some constantc. Of course this is an asymptotic notion, where we think ofG as ranging overan infinite family of groups, whereas the constantc does not depend onG. Here we focus on thecaseG = Sn, where distinguishability is equivalent toDH ≥ n−c. Distinguishable subgroupsHare those which can be distinguished from 1 using the so called weak standard method (see the nextsection for more details).

The main application of this paper to Quantum Computing, which relies heavily on Theorem Babove, is the following.

Theorem C. LetH ≤ Sn be a subgroup. IfH is distinguishable, then it has a bounded minimaldegree. Moreover, ifDH ≥ n−c, thenm(H) ≤ g(c), whereg(x) = ax + b is some fixed linearfunction.

Thus all subgroups of unbounded minimal degree are indistinguishable, which opens up a hugespectrum of examples and constructions. The only case previously known in the literature of anindistinguishable subgroup ofSn is that of a subgroup of order2 generated by a fixed point free

4

involution or by a product of transpositions of large support [HRT00, GSVV01]. Obviouslym(H)is unbounded for such subgroupsH, so its indistinguishablity is an immediate consequence oftheabove theorem.

In an extended abstract [KS05] a subset of the authors of thispaper have proved a weaker versionof Theorem C (for primitive subgroups and subgroups of polynomial size) and have conjectured thatit holds in full generality. This paper proves the conjecture.

It is intriguing that much larger subgroups are also indistinguishable. Indeed takeH = S2n/m <Sn, the subgroup constructed following Theorem A. Ifm = m(H) tends to infinity arbitrarilyslowly, thenH is indistinguishable and|H| ≥ (n!)ε(n) whereε(n) tends to0 arbitrarily slowly. Inparticular, the size of indistinguishable subgroups ofSn can be super-exponential inn.

However, ifε > 0 is fixed, and|H| ≥ (n!)ε, then it follows from Theorem A that the minimaldegree ofH is bounded. Enumerating over elements ofSn of bounded support (their number isbounded by a polynomial inn) we deduce that such a subgroupH can be distinguished from 1using classical search.

It follows from the two paragraphs above thatall subgroupsH ≤ Sn of size≥ N can bedistinguished from 1 using the weak standard method (together with classical search) if and only ifN ≥ (n!)ε whereε is bounded away from zero.

Theorem C has rather grave consequences. Indeed, ifH is distinguishable then it has an elementof bounded support, and this can be detected (as above) afterpolynomially many membership tests(when we enumerate the permutations inSn according to their support).

Corollary D. Any subgroupH ≤ Sn which is distinguishable can already be distinguished from1using classical search.

Thus Theorem C provides a complete characterisation of hidden subgroupsH ≤ Sn which canbe distinguished from 1 using the weak standard method and classical search: these are preciselythe subgroups of bounded minimal degree.

It is intriguing that the old classical notion of minimal degree, which is central in the theory offinite permutation groups, plays a role in the context of quantum computing. The Classification ofFinite Simple Groups (CFSG) is also used in an essential way in some parts of this work.

Some words on the structure of this paper. In Section 2 we provide background on quantumcomputing, the Hidden Subgroup Problem, and the standard method of Quantum Fourier Sampling.Section 3 deals with arbitrary finite groupsG and their subgroupsH. Using character-theoreticmethods we give upper and lower bounds on theL1-distanceDH introduced above. We then charac-terize distinguishable subgroups of polylogarithmic size. In Section 4 we focus on the caseG = Sn.We prove there (relying on CFSG and other tools) that any primitive subgroupH < Sn not contain-ing An is indistinguishable. We also show how to deduce Theorem C from Theorem B. TheoremA is proved in Section 5. Section 6, which is the longest in this paper, is devoted to the proof ofTheorem B. This proof applies Theorem A as well as results on primitive groups obtained in Section4.

5

2 Quantum Computing

In the last decade quantum computation has provided us with powerful tools to solve problems notknown to be classically efficiently solvable, like factoring and discrete log [Sho94]. Nearly all theproblems in which a quantum computer excels more than quadratically with respect to its classicalcounterpart can be cast into the framework of the Hidden Subgroup Problem (HSP). LetG be afinite group andH ≤ G a subgroup. Given a functionf : G → S that is constant on (left)-cosetsgH of H and takes different values for different cosets, determinea set of generators forH. Thedecision version of this problem is to determine whether there is a non-identity hidden subgroup ornot.

The reason that quantum computers seem to provide a speed-upfor this type of problem is thatit is possible to implement the Fourier transform over certain groupsefficientlyon a quantum com-puter. This in turn allows to sample the Fourier components efficiently (this technique of QuantumFourier Sampling is referred to as the “standard method”). In the case of Abelian groupsG (appear-ing in factoring and discrete log) the hidden subgroup can bereconstructed with only a polynomial(in log |G|) number of queries to the function and a polynomial number ofmeasurements (samplingsin the Fourier basis) and postprocessing steps.

We denote states of the vector spaceC[G], spanned by the group elements, with a|·〉, as isstandard in quantum computation (see e.g. [NC00] for more details).

Definition 1. The Quantum Fourier Transform (QFT) over a groupG is the following unitarytransformation onC[G]:

|g〉 → 1√|G|

∑

ρ,i,j

√dρρ(g)ij |ρ, i, j〉

whereρ labels an irreducible representation ofG, dρ is its dimension and1 ≤ i, j ≤ dρ. The|ρ, i, j〉 span another basis ofC[G], the so called Fourier basis.

For many non-Abelian groups it is possible to implement the Fourier transform on a quan-tum computer efficiently, and in particular explicit constructions exist for the symmetric groupSn

[Bea97].Addressing the HSP in the non-Abelian case is considered to be one of the most important

challenges at present in quantum computing. A positive answer to the question whether quantumcomputers can efficiently solve the Hidden Subgroup Problemover non-Abelian groups would haveseveral important implications for the solution of problems in NP, which are neither known to beNP-complete nor in P; and which are good candidates for a quantum speed-up. Among the mostprominent such problems is Graph Isomorphism, where the group in question is the symmetricgroup. Hence it is very desirable to get a handle on the power of Quantum Fourier Sampling (QFS)to solve the HSP for general groups.

Definition 2. The standard method of Quantum Fourier Sampling is the following: The state isinitialised in a uniform superposition over all group elements; a second register is initialised to|0〉.Then the functionf is applied reversibly over both registers (i.e.f : |g〉|0〉 → |g〉|f(g)〉). Thesecond register is measured, which puts the first register into the superposition of a (left)-coset ofH, i.e. in the state|gH〉 := 1√

|H|∑

h∈H |gh〉 for some randomg ∈ G. Finally the QFT overG is

6

performed, yielding the state

1√|G||H|

∑

ρ,i,j

√dρ∑

h∈Hρij(gh)|ρ, i, j〉.

A basis measurement now gives(ρ, i, j) with probabilityPgH(ρ, i, j) =dρ

|G||H| |∑

h∈H ρij(gh)|2.

Since we do not knowg and g is distributed uniformly, we sample(ρ, i, j) with probabilityPH = 1

|G|∑

g PgH . Thestrongstandard method samples bothρ and its entriesi, j. In theweakstandard method only the characterχρ is measured (but not the entriesi, j, which are averaged over).In this case it is not hard to see [HRT00, GSVV01] that the probability to sampleρ is independentof the coset ofH we happen to land in. Hence the probability to measureρ in the weak case is

PH(ρ) =dρ|G|

∑

h∈Hχρ(h).

Note that from this expression it is clear that the weak standard method cannot distinguish betweenconjugate subgroups [HRT00]. LetIrr(G) be the set of irreducible characters ofG. ThenPH

is a distribution onIrr(G). The strong standard method sometimes provides substantially moreinformation than its weak counterpart, and is indeed necessary to efficiently solve the HSP in thecase of groups like the Dihedral group [EH99, Kup03, Reg04] and other semidirect product groups[MRRS04]. However (see below), forSn Grigni et al. [GSVV01] have shown that for arandomba-sis the additional information provided by the strong method is exponentially small except possiblyfor very large subgroups.

An even more basic question is which hidden subgroups can bedistinguished from the identityvia QFS with special attention to the symmetric group. Distinguishing the trivial subgroupefrom a larger subgroupH efficiently using the weak standard method is possible if andonly if theL1 distanceDH betweenPe andPH is larger than some inverse polynomial inlog |G|. TheL1

distance (also known as the total variation distance) is given as

DH =1

|G|∑

ρ

dρ|∑

h∈H,h 6=1

χρ(h)|.

We say thatH is distinguishable(using the weak standard method) ifDH ≥ (log |G|)−c for someconstantc, andindistinguishableotherwise.

Several positive results on the power of QFS for the Hidden Subgroup Problem have been ob-tained previously for groups that are in some ways “close” toAbelian, like some semidirect prod-ucts of Abelian groups [EH99, RB98, Kup03, Reg04, MRRS04], in particular the Dihedral group;Hamiltonian groups [HRT00], groups with small commutator groups [IMS01] and solvable groupsof constant exponent and constant length derived series [FIM+03]. Often in these cases the irre-ducible representations are known and can be analysed. For instance the Dihedral groupDn, thefirst non-Abelian group to be analysed in this context by Ettinger and Hoyer [EH99], is “nearly”Abelian in the sense that all of its irreducible representations have degree at most two. Indeed hid-den reflections ofDn can be distinguished from the identity with only polynomialQuantum FourierSamplings, similar to the Abelian case (where all irreducible representations are one-dimensional).

7

Note, however, that the computational version of the HSP seems much harder: even though a poly-nomial number of samples suffice todistinguishhidden reflectionsinformation theoretically, noefficient reconstruction procedure is known.

The holy grail of the field is the symmetric groupSn, which seems much harder to analyse,partly because to this day there is still only partial explicit knowledge about its irreducible represen-tations and character values [Sag01], because most of its subgroups are far from normal (have manyconjugate subgroups), because most of its irreducible representations have very large dimension(2Θ(n logn)) and the number of different irreducible representations is an exponentially small frac-tion of the size of the group, to name just some of the difficulties. The structure of distinguishableversus indistinguishable subgroups ofSn has remained open.

The following results have been obtained for the HSP over thesymmetric group: The groupSn being non-Abelian, Quantum Fourier Sampling gives a distribution on both the characters andthe entries of the corresponding matrix representations. Grigni et al. [GSVV01] show that sam-pling the row index in the strong standard method provides noadditional information. They alsoshow that the additional information provided by the strongmethod in arandombasis scales with3√

|H|2k(G)/|G| wherek(G) is the number of conjugacy classes of the groupG and|H| the sizeof the hidden subgroup. Both Hallgren et al. and Grigni et al.[HRT00, GSVV01] show that hiddensubgroups ofSn of size |H| = 2, generated by involutions with large support, cannot be distin-guished from identity; exactly the task that needs to be solved for Graph Automorphism. Recently,Moore et al. have essentially shown that thestrongstandard method cannot distinguish the subgroupgenerated by a fixed point free involution from identity [MRS05]. Moreover, even a generalizationof the strong standard method toO(n log n) instances of Quantum Fourier Sampling does not allowto distinguish the above subgroup from1 [H+06]. No results are known for other subgroups ofSn.

In this work various classical as well as modern parts of the theory of permutation groups areapplied for the first time in the context of quantum computing. In our applications to the hiddensubgroup problem, we focus on theweakform of the standard method, since the strong form withrandom choices of basis does not provide any non-negligibleadditional information for the sym-metric group and the subgroups we consider [GSVV01]. It remains to be seen whether judiciouschoices of basis for each irreducible matrix representation can give more information in the casewhere random choices don’t help; but to our knowledge no suchexamples have been found and infact recent results of Moore et al. [MRS05] show that in the case of fixed point free involutions nosuch good basis exists.

Theorem C and Corollary D above provide a complete characterization of subgroups whichcan be distinguished from1 using the weak standard method (together with classical exhaustivesearch). Indeed, these are exactly the subgroups ofSn with bounded minimal degree. For instancewe cannot distinguish a group generated by a cycle of unbounded length or an involution withunbounded number of transpositions (implying the result in[HRT00, GSVV01]).

This also has implications for the Graph Isomorphism (GI) problem. Recall that to solve GIfor two graphsG1, G2, it suffices to distinguish a hidden subgroup of the automorphism groupAut(G1 ∪ G2) of the formH1 × H2 (not G1 ≃ G2), whereHi = Aut(Gi), from a subgroupof the formH ∪ σH (G1 ≃ G2), whereH = H1 × H2 and σ mapsG1 to G2. Our resultsimply that we cannot distinguish each of the two possible cases from identity, and hence (using thetriangle inequality) we cannot distinguish them from each other unless Aut(Gi) contains an elementof bounded support. Thus weak QFS provides no advantage here.

8

3 Arbitrary groups

In this section we discuss results for arbitrary finite groupsG. Our starting point is a general resultproviding both upper and lower bounds on the total variationdistanceDH in terms of the samegroup theoretic data. While the definition ofDH involves character degrees and values, which arehard to compute, our bounds below involve sizes of conjugacyclasses, and their intersections withthe hidden subgroup.

We need some group theoretic notation. Forh ∈ G we lethG denote the conjugacy class ofhin G. LetC1, . . . , Ck denote the non-identity conjugacy classes ofG. For an irreducible characterχρ ∈ Irr(G) we letχρ(Ci) denote the common value ofχρ(x) for elementsx ∈ Ci.

Proposition 1. LetH < G. Then

1.k∑

i=1

|Ci ∩H|2|H|−1|Ci|−1 < DH

2. DH ≤k∑

i=1

|Ci ∩H||Ci|−12 =

∑

16=h∈H|hG|−1/2.

Applying the upper bound with|H| = 2 gives the result obtained previously by Hallgren etal. and Grigni et al. [HRT00, GSVV01]. No lower bounds seem toexist in the literature. Thisresult has a wide range of applications. For example, it enables us to characterise distinguishablesubgroupsH ≤ G of polylogarithmic order (see Theorem 3 below).

Proof of Proposition 1.For each irreducible representationρ of G we have

|∑

h∈H,h 6=1

χρ(h)| ≤∑

h∈H,h 6=1

|χρ(h)| ≤∑

h∈H,h 6=1

dρ < |H|dρ.

Hencedρ > |H|−1|∑

h∈H,h 6=1 χρ(h)|. Substituting this in (1) we obtain

DH >1

|G||H|∑

ρ

|∑

h∈H,h 6=1

χρ(h)|2.

Note thatχρ(h) = χρ(Ci) if h ∈ H ∩ Ci. This yields∑

h∈H,h 6=1 χρ(h) =∑k

i=1 |H ∩ Ci|χρ(Ci),and so

DH >1

|G||H|∑

ρ

|k∑

i=1

|H ∩ Ci|χρ(Ci)|2.

Now,

|k∑

i=1

|H ∩ Ci|χρ(Ci)|2 =k∑

i=1

|H ∩ Ci|2|χρ(Ci)|2 +∑

i 6=j

|H ∩Ci||H ∩ Cj|χρ(Ci)χρ(Cj).

Using the generalised orthogonality relations we observe that

∑

ρ

k∑

i=1

|H ∩ Ci|2|χρ(Ci)|2 =k∑

i=1

|H ∩ Ci|2|G|/|Ci|,

9

and ∑

ρ

∑

i 6=j

|H ∩ Ci||H ∩Cj |χρ(Ci)χρ(Cj) = 0.

It follows that

DH >1

|G||H|

k∑

i=1

|H ∩ Ci|2|G|/|Ci| =k∑

i=1

|H ∩Ci|2|H|−1|Ci|−1.

This completes the proof of the lower bound.To prove the upper bound, write

DH |G| =∑

ρ

dρ|∑

h∈H,h 6=1

χρ(h)| ≤∑

ρ

dρ∑

h∈H,h 6=1

|χρ(h)| =∑

h∈H,h 6=1

∑

ρ

dρ|χρ(h)|. (2)

Fix h ∈ H and choosei such thath ∈ Ci. Using the Cauchy-Schwarz inequality we obtain∑

ρ

dρ|χρ(h)| ≤ (∑

ρ

d2ρ)1/2(

∑

ρ

|χρ(h)|2)1/2,

giving (using the orthogonality relations)∑

ρ

dρ|χρ(h)| ≤ |G|1/2(|G|/|Ci|)1/2 = |G||Ci|−1/2.

Summing over non-identity elementsh ∈ H, and observing that the upper bound above occurs|H ∩ Ci| times, we obtain

∑

h∈H,h 6=e

∑

ρ

dρ|χρ(h)| ≤k∑

i=1

|H ∩ Ci||G||Ci|−1/2.

Combining this with (2) we obtain

DH ≤k∑

i=1

|H ∩ Ci||Ci|−1/2,

as required.

The following is an immediate consequence of Proposition 1.

Corollary 2. LetCmin denote a non-identity conjugacy class of minimal size intersectingH non-trivially. Then we have

|H|−1|Cmin|−1 < DH ≤ (|H| − 1)|Cmin|−1/2.

We can now prove the main result of this section, characterising distinguishable subgroups ofpolylogarithmic order in an arbitrary groupG.

Theorem 3. Suppose|H| ≤ (log |G|)c for some constantc. ThenH is distinguishable if and onlyif H has a non-identity elementh such that|hG| ≤ (log |G|)c′ for some constantc′.

10

Proof.Suppose first thatH is distinguishable, namelyDH ≥ (log |G|)−b for some constantb. Then

the upper bound in the above corollary shows that

|H||Cmin|−1/2 ≥ (log |G|)−b,

so|Cmin| ≤ |H|2(log |G|)2b) ≤ (log |G|)2(b+c).

In the other direction, suppose|Cmin| ≤ (log |G|)b. Then the lower bound in the corollary abovegives

DH > |H|−1(log |G|)−b ≥ (log |G|)−(b+c).

The result follows.

4 Symmetric groups

Let us now focus on the caseG = Sn. In this section we first prove some preliminary results relatedto distinguishability of subgroups ofSn. Some of these results play a role in the proof of TheoremB. We also deduce Theorem C from Theorem B.

Proposition 4. LetH ≤ Sn with |H| ≤ nc for some constantc. ThenH is distinguishable if andonly if its minimal degreem(H) is bounded.

Proof. Let g ∈ Sn with supp(g) = k. Then it is straightforward to verify that(nk

)≤ |gSn | ≤ nk. As

a consequence we see that a conjugacy classC in Sn has polynomial order if and only if it consistsof elements of bounded support. This observation, when combined with Theorem 3, completes theproof.

Our next result concerns primitive subgroups. Primitive permutation groups are considered thebuilding blocks of finite permutation groups in general, andwere extensively studied over the past130 years. We note that ifH ≤ Sn is primitive andH 6= An, Sn then Babai showed that|H| ≤n4

√n logn. Using the Classification of Finite Simple Groups the latterbound can be somewhat

improved to|H| ≤ 2n√n, which is essentially best possible [Cam81]; in particularthe order ofH

can be much more than polynomial, and so Proposition 4 above does not apply.However, we obtain the following somewhat surprising general result:

Theorem 5. Let H 6= An, Sn be a primitive subgroup. ThenH is indistinguishable. Moreover,there is an absolute constantε > 0 such that

DH ≤ n−ε√n.

This theorem follows immediately from the two technical lemmas below, which are based oncounting elements of given support in permutation groupsH. Recall that forH ≤ Sn we set

Hk = h ∈ H : |supp(h)| = k.

11

Lemma 6. LetH ≤ Sn be a subgroup. Suppose that, for eachk ≤ n, we have

|Hk| ≤ n(1/6−ε)k.

whereε > 0 is some fixed constant. Then, ifn is large enough (givenε) we have

DH ≤ 2n−δm(H),

whereδ = ε/2. In particular, if the minimal degreem(H) is unbounded, thenH is is indistinguish-able.

Proof: Apply the upper bound of Proposition 1, written in the form

DH ≤∑

16=h∈H|hG|−1/2.

To evaluate this sum we use a result from [LSh01], showing that, for G = Sn andh ∈ G of supportk we have|hG| > nak for any reala < 1/3 andn large enough (givena). Using this we obtain

DH <∑

k≥m(H)

|Hk|n−bk,

for any real numberb < 1/6 and sufficiently largen. Let δ = ε/2, b = 1/6 − δ, andm = m(H).Then the upper bound on|Hk| yields

DH <∑

k≥m

n(1/6−ε)kn−(1/6−δ)k =∑

k≥m

n−δk ≤ 2n−δm.

This proves the first assertion. Assumingm = m(H) is unbounded, we see thatDH is smaller thanany fixed negative power ofn, and soH is indistinguishable.

Lemma 7. LetH < Sn be primitive andH 6= An, Sn. Then for sufficiently largen and for all kwe have|Hk| ≤ n

k7 .

Proof: We use Babai’s lower bound on the minimal degree of primitivesubgroupsH 6= An, Sn

[Bab81], showing thatm(H) ≥ (

√n− 1)/2. (3)

Furthermore, we apply a theorem of Cameron [Cam81] (which inturns relies on the Classificationof Finite Simple Groups) describing all primitive groups of‘large’ order. In particular it followsfrom that description that, for all largen, and for a primitive subgroupH 6= An, Sn, either

(i) |H| ≤ ncn1/3, or

(ii) n =( l2

)for somel, andH ≤ Sl acting on2-subsets of1, . . . , l, or

(iii) n = l2 for somel, andH ≤ Sl ≀ S2 acting on1, . . . , l2 in the so called product action.We claim that for all largen and for allk we have|Hk| ≤ nk/7. To show this it suffices to

considerk ≥ (√n − 1)/2, otherwise|Hk| = 0 by (3). Now, if H satisfies condition (i) above

then the claim follows trivially using|Hk| ≤ |H|. So it remains to consider groupsH in cases (ii)

12

and (iii). Here a simple computation based on the known actions ofH completes the proof of theLemma.

Theorem 5 now follows by combining the above two lemmas. In fact we obtain, for all primitivesubgroupsH 6= An, Sn,

DH ≤ 2n−m(H)/84 ≤ 2n−(√n−1)/168.

The remainder of this section is devoted to reducing TheoremC to Theorem B.

Lemma 8. Let C be a conjugacy class inSn consisting of elements of supportk. Then|C| ≥c(nk

)√k! · k− 1

2 , wherec is an absolute positive constant.

Proof: There a(nk

)ways to chose the subsetS ⊂ 1, . . . , n of letters moved by an element

h ∈ C. Given the subsetS, h|S is a fixed point free permutation of degreek. The number of suchpermutations with a given cycle structure is minimal in the case of a fixed point free involution andis in this case equal tok!/2

k2 (k/2)!. Using Stirling’s formula, we see that this expression is atleast

c√k! · k− 1

2 . Putting everything together the lemma follows.

Lemma 9. LetH ≤ Sn. Then

DH ≤ a∑

1≤k≤n

|Hk|(n

k

)− 12

(k!)−14 · k 1

4 ,

wherea is some absolute constant.

Proof: We use part 2 of Proposition 1:

DH ≤∑

16=h∈H|hG|−1/2.

By Lemma 8 we conclude that

∑

h∈Hk

|hG|− 12 ≤ c−1/2|Hk|

(n

k

)− 12

(k!)−14 · k 1

4 .

The result follows.Suppose now that Theorem B holds and letm = m(H). Substituting

|Hk| ≤ n−εm

(n

k

) 12

(k!)14

in Lemma 9 we obtainDH ≤ an−εm

∑

1≤k≤n

k14 ≤ an−εm · n 5

4 .

Therefore, ifm is unbounded,DH is smaller than any inverse polynomial inn, and henceH isindistinguishable. Moreover, assumingDH ≥ n−c (andn3/4 ≥ a as we may) we obtainεm−2 ≤ c,and so

m ≤ 2/ε + c/ε.

Hence Theorem C follows from Theorem B.

13

5 Bounds on the group size in terms of the minimal degree

In this section we prove Theorem A. It extends a theorem of Martin Liebeck [L82, L84] whichbounds the order of transitive groups with large minimal degree.

We callH asubdirect product subgroupof St if it is a subdirect product ofS1× · · · ×St whereall theSi are isomorphic toS. Such anH is called adiagonal subgroupif it is isomorphic toS.

Lemma 10. Let S be a non-abelian simple group andH a subdirect product subgroup ofSt ∼=S1 × · · · × St.

1) Then there is a partition of the set of indices1, . . . , t and for each part, sayij1 , . . . , ijk,a diagonal subgroupDj of Sij1

× · · · × Sijksuch thatH is a direct product of the subgroupsDj .

2) Assume thatS ∼= Alt(k) for somek ≥ 7 and letD be a diagonal subgroup ofSt. Letd = (d1, . . . , dt) be an element ofD such thatd1 is a3-cycle. Then all thedi are3-cycles.

Proof. 1. This is a standard result.2. This follows from the fact that the set of3-cycles is invariant under automorphisms of Alt(k)

if k ≥ 7 [DM96, Lemma 8.2. A].

Let H be a permutation group with minimal degreem = m(H). Denote byΩ1, . . . ,Ωr theorbits ofH and sett = max |Ωi|. Let Bi = Bi1 , . . . , Biki

a system of blocks of imprimitivityfor the action ofH onΩi such thatki > 1 is minimal (if H acts onΩi as a primitive group, thenki = |Ωi|). Denote byKi the kernel of the action ofH onBi and the size of the blocks inBi by bi.

SetB =r⋃

i=1Bi, K =

r⋂i=1

Ki andx =r∑

i=1(ki − 1). Note thatK has at leastr + x orbits.

Proposition 11. |H/K| ≤ 5x t3n/m.

Proof. H acts onBi as a primitive permutation groupPi∼= H/Ki of degreeki. If Pi does not

contain Alt(ki), then, by a result of Praeger and Saxl, [PS80] we have|Pi| ≤ 4ki . Together withsome trivial computation for small values ofki this implies|Pi| ≤ 5ki−1.

Denote byS the intersection of all theKi for which |Pi| ≤ 5ki−1 holds. ThenS acts on eachBi

either as a trivial group or as a group containing Alt(ki) whereki ≥ 7. Without loss of generalityone can assume thatS acts trivially onBi exactly if i > q. The groupA = (S/K)′ is a subdirectproduct subgroup of Alt(k1) × · · · × Alt(kq). Denoting byA the inverse image ofA in S we seethat |H/A| ≤ 5x holds.

To complete the proof it is enough to show that

|A/K| = |A| ≤ t3n/m.

It follows from Lemma 10 thatA is a direct product of diagonal subgroupsAj. EachAj acts asan alternating group Alt(nj) on some systems of blocksBi with nj = ki, trivially on the rest and isisomorphic to Alt(nj).

We claim that the sum of the block-sizesbi corresponding toAj is at leastm/3. To simplifynotation we assume thatAj acts trivially onBi exactly if i > p. By Lemma 10 there is an elementaj of Aj which acts as a3-cycle on eachBi for i ≤ p. This element corresponds to an elementaj

of A which moves at most3p∑

i=1bi elements. Hence3

p∑i=1

bi ≥ m as claimed.

14

It follows that eachAj moves at leastnjm/3 points. This implies that the sum of thenj for alldiagonal subgroupsAj is at most3n/m. EachAj has order12nj! ≤ tnj . Hence|A| ≤ t3n/m asrequired.

We are now ready to prove Theorem A:

Proof of Theorem A.Set ℓ = min(m, log2 n). We have to show that|G| ≤ n10nℓ . Denote by

∆1, . . . ,∆t the orbits ofG. Let Di = Di1, . . . ,Dihi be a system of blocks of imprimitivity for

the action ofG on∆i, such that the block sizedi is at leastℓ anddi is as small as possible with this

restriction (if there are no proper blocks of size≥ ℓ then we setDi1 = ∆i). G acts onD =t⋃

i=1Di

as a permutation group of degree at mostn/ℓ. Hence the kernelH of the action has index≤ nnℓ

in G.Denote byΩ1, . . . ,Ωr the orbits ofH and letB1, . . . ,Br be systems of imprimitivity as in

Proposition 11. By the construction ofH it is clear that we havebi < ℓ for eachi. ApplyingProposition 11 we obtain a subgroupK of index≤ 5xn3n/m such thatK has at leastr + x orbitsand each orbit has size< ℓ.

We apply Proposition 11 toK to obtain a subgroupK1 of index≤ 5x1 · ℓ3n/m in K, which hasat leastr + x+ x1 orbits, each of size≤ ℓ

2 .Continuing in this fashion we obtain a descending series of subgroupsK > K1 > K2 > · · · >

Kv = 1. The maximal size of an orbit ofKi is at mostℓ/2i, hence the above series of subgroupshas lengthv ≤ log2 ℓ.

SinceKi has at leastr + x + x1 + · · · + xi orbits we havex + x1 + · · · + xv ≤ n. Hence

|H| = |H/K| · |K/K1|v−1∏i=1

|Ki/Ki+1| ≤ 5nn3n/m · (ℓ3n/m)v ≤ 5n · n3n/ℓ · 23n

((log ℓ)2

ℓ

)

≤

5nn3n/ℓ · 23n·9/8 ≤ n3n/ℓ · 26n. Therefore we have|G| ≤ n4n/ℓ · 26n ≤ n10nℓ as required.

6 Counting elements of given support

This section, which is the longest in this paper, is devoted to the proof of Theorem B. The mainingredients of the proof are Theorem A and Proposition 5.

We will use the following inequality many times.

Proposition 12. Letx, y, n be positive integers such thatx+ y ≤ n. Then(nx

)(ny

)≤( nx+y

)22(x+y)

holds.

Proof. In fact we claim that the stronger inequality(nx

)(ny

)≤(

nx+y

)(x+yy

)2holds. This is equivalent

ton(n− 1) . . . (n− x+ 1)n(n − 1) . . . (n− y + 1)

n(n− 1) . . . (n − x− y + 1)≤(x+ y

y

)

which is equivalent to

n(n− 1) . . . (n− y + 1)

(n − x) . . . (n− x− y + 1)≤ (x+ y)(x+ y − 1) . . . (x+ 1)

y!

15

But this follows by multiplying the inequalities

n− t

n− x− t≤ x+ y − t

y − tfor t = 0, 1, . . . , y − 1.

These latter inequalities follow fromx+ y ≤ n.

To avoid some technical difficulties we first prove Theorem B directly in the case whenk is verylarge.

Lemma 13. Let H be a permutation group of degreen and minimal degreem ≥ 100 000. As-sume thatk ≥ n

23+ 1

100 and k ≥ 2100 000. Then there exists a constantε > 0 such that|Hk| ≤(nk

) 12 (k!)

14n−εm holds.

Proof. We have to count elementsh ∈ H with supp(h) = k. There are at most(nk

)choices for

supp(h) and given this by Theorem A there are at mostkk

10 000 choices forh itself. We have to showthat (

n

k

)k

k10 000 ≤

(n

k

) 12

(k!)14n−εm.

This is equivalent to (n

k

)k

k5000 · n2εm ≤ (k!)

12

which follows fromnk · k k

5000 · n2εk ≤ (k!)32 .

This in turn is implied by

nk · k k5000 k3εk ≤

(k

e

) 32k

which reduces ton

23(e · k 1

7500+2ε)≤ k

which follows from our conditions ifε is small enough.

We now fixa ≥ 10 000 such that ifH is a primitive permutation group of degreen ≥ a notcontaining Alt(n), thenm(H) ≥ 100 and|Hk| ≤ nk/7. This is possible by [Bab81] and Lemma 7above.

Next, we introduce some notation which will be used in the rest of this section. LetG be apermutation group of degreen with no fixed points. Denote byΩ1,Ω2, . . . , the orbits ofG. LetBi = Bi1, Bi2, . . . be a system of blocks of imprimitivity for the action ofG on Ωi, such that|Bi1| ≥ 2 is minimal. Then the setwise stabiliser of the blocksBij in G acts as some primitivegroupPij onBij . ThePij are permutation equivalent fori fixed.

We partition the set of blocksB =⋃Bi into 3 subsets as follows. Denote byS = S1, S2, . . .

the set of blocks of size< a. Denote byA = A1, A2, . . . the set of blocksBij in B \ S forwhich Pij contains Alt(Bij), and denote byL = L1, L2, . . . , the set of the remaining blocks.SetS =

⋃Si, A =

⋃Ai andL =

⋃Li. It is clear that anyg ∈ G fixes the setsS, L andA. We

denote the action ofg ∈ G on a setX (fixed byg) by gX and the action ofG on a fixed setX byGX .

Our next lemma shows that in a sense there are not too many possibilities for the action of someg ∈ G on the setS ∪ L.

16

Lemma 14. 1)The number of pairs(supp(gS), gL) for permutationsg with |supp(g)| = k, |supp(gL)| =x and |supp(gA)| = y is at most

(n

[ k−x−y2 ]

)2ak · nx( 1

7+ 1

100 ).

2) Givensupp(gS), the number of possible actionsgS is at mostak−x−y[k−x−y

2

]!. In fact this

is an upper bound for the number of possible actions onsupp(gS) of elementsh which fixsupp(gS).

Proof. If g moves a point of some block, then it moves at least two points of the block. Hence the

numbert of blocks inS which contain points from supp(g) is at most[k−x−y

2

]. These blocks can

be chosen in at most(nt

)ways. Given these blocks the number of choices for supp(gS) is at most

(2a)t.Note thatgS (or h ∈ G fixing supp(gS)) movesa1, a2, . . . , at given points of the chosen blocks

in at mosta1!a2! . . . at! · t! ≤t∏

i=1aai · t! ≤ ak−x−y

[k−x−y

2

]! ways, proving 2).

Each block inL which contains points of supp(g) contains at least 100 such points (by thechoice ofa, see the notation introduced after Lemma 13), hence the number ℓ of such blocks is atmostx/100. These blocks can be chosen in at most

(nℓ

)≤ n

x100 /ℓ! ways.

There areℓ1 ≤ ℓ blocks fromL fully contained in supp(g) and these can be chosen in at most2ℓ ways.

By our assumption on the blocks inL and the Praeger–Saxl theorem [PS80] the stabilisers ofa blockBij in L can act on the block in at most4|Bij | ways. This implies that the stabiliser of theunion of the above blocks can act on this union in at most4xℓ1! ways. Hence this is an upper boundfor the number of actions ofg on the blocks contained in supp(g).

Assume that on the remaining blocks (which are as sets fixed byg) g acts as a permutation ofdegreex1, x2, . . . . The numberx1, x2, . . . can be chosen in at most2x ways. Given these numbersthe number of actions ofg on these remaining blocks can be chosen in at mostnx1/7 · nx2/7 · · · ≤nx/7 ways by Lemma 7.

Altogether the number of choices for supp(gS) andgL is at most

(n

t

)2at(n

x100 /ℓ!

)2ℓ4xℓ1!2

x · nx/7 ≤(

n[k−x−y

2

])n

x7+ x

100 2ak

as required.

Corollary 15. The number of pairs(supp(gS), gL) for permutationsg with |supp(g)| = k and|supp(gA)| = y is at most

(n

k

) 12 [y

2

]!n− y

2 · 2(a+4)k if k ≤ n23+ 1

100

andn is sufficiently large.

Proof. We first claim that the number of permutationsg considered is at most1n

(n

[k−y2 ]

)2(a+1)k.

By Lemma 14 it is sufficient to prove that for allx ≤ k we have(

n[k−x−y

2

])2aknx(1

7+ 1

100) ≤ 1

kn

(n[

k−y2

])2(a+1)k .

17

This is obvious ifx = 0, otherwise we havex ≥ 100. By Proposition 12(

n[k−x−y

2

])(

n[x2

])

≤(

n[k−y2

])2k

holds, hence it is enough to show thatnx(17+ 1

100)+2 ≤(

n[ x2 ]

). But this follows using100 ≤ x ≤

k ≤ n23+ 1

100 .Using Proposition 12 we obtain that

1

n

(n[

k−y2

])2(a+1)k ≤ 1

n

(n[k2

])(

n[y2

])−1

2(a+2)k

≤ 1

n

(n

k

) 12(

n[y2

])−1

2(a+3)k ≤ 1

n

(n

k

) 12 [y

2

]!n−[ y2 ]2(a+4)k

≤(n

k

) 12 [y

2

]!n− y

2 2(a+4)k

proving the corollary.

The most difficult part of the proof of Theorem B is wheny is large compared tom. Thefollowing result implies Theorem B in the case when this holds and moreoverk! is large comparedto ny.

Lemma 16. Assume thatm ≥ 100 000, k ≤ n23+ 1

100 , n3y ≤ k! and k is sufficiently large (inparticular k ≥ 2100 000). Then the number of permutationsg with |supp(g)| = k and|supp(gA)| = y

is at most(nk

) 12 (k!)

14n− y

60 .

Proof. The number of choices for supp(gA) is at most(ny

). Hence by Corollary 15 the number of

choices for supp(g) is at most

(n

k

) 12 [y

2

]!n− y

2 2(a+4)k

(n

y

)≤(n

k

) 12

ny2 2(a+4)k.

Using Theorem A we see that the number of choices forg is at most

(n

k

)12

ny2 2(a+4)kk

k10 000 ≤

(n

k

) 12

(k!)15 · k k

10 000 · 2(a+4)k · n− y60 .

If k is large enough (compared to the constanta) then(k!)120 ≥ k

k10 000 ·2(a+4)k and our statement

holds.

Next we describe an important subgroup ofG. Consider the set consisting of the points inSandL and the blocks inA. LetK be the kernel of the action ofG on this set. By definitionK fixesall the points outsideA. Moreover, ifAi ∈ A, then the actionKi of K onAi is a normal subgroupof the action of the stabiliser ofAi in G, hence it is either Sym(Ai), Alt(Ai) or 1.

18

Without loss of generality one can assume thatK acts trivially onAi exactly if i > q. NowK is a subdirect product of theKi, therefore its commutator subgroupK ′ is a subdirect productsubgroup of Alt(A1) × · · · × Alt (Aq). Hence by Lemma 10K ′ is a direct product of diagonalsubgroupsDj. EachDj acts as an alternating group Alt(nj) on some blocksAi of sizenj. ByLemma 10Dj contains an elementdj which acts as a 3-cycle on each of the correspondingAi.HenceDj acts non-trivially on at leastm3 blocksAi (since|supp(dj)| ≥ m). NowK is a subgroup

of the normaliserN of K ′ inq∏

i=1Sym(Ai). ClearlyN is a direct product of groupsNj ≥ Dj where

Nj is isomorphic to Sym(nj) and containsDj∼= Alt(nj) in a natural way.

Proposition 17. There are at mostn3hm elementsg ofK with |supp(g)| = h (wherem = m(G)).

Proof. We have a unique decompositiong = g1g2 . . . wheregj ∈ Nj. Let us choose for eachj ablock on whichNj acts non-trivially. It is clear thatgj is determined uniquely by its action on thechosen block. Thereforeg is determined by its action on the unionU of the chosen blocks.

It follows by the above discussion that|supp(g) ∩ U | ≤ 3hm . Hence the number of choices forg

is at most|U | 3hm ≤ n3hm .

Proposition 18. Assume thatm ≥ 100 000. Then the number of permutationsg with gS∪L fixedandsupp(gA) = y is at mostny/5000.

Proof. The cosetgK is determined bygS∪L and the action ofg on the blocks inA. Now g canmove at mostt ≤ y

a blocks inA.

The number of choices for these blocks is less than(n/a

t

)and given these blocks the number of

waysg can act on them is at mostt!. Henceg can act in at most(na

)[ ya ] +(na

)[ ya ]−1+ · · · ≤ n

ya

ways onA. If gK contains another elementf with |supp(f)| = k and |supp(fA)| = y, then

gf−1 ∈ K and|(supp(gf−1)| ≤ 2y. Hence by Proposition 17 there are at mostn6ym ≤ n

y10 000 such

elementsgf−1. Of courseg andgf−1 determinesf . Altogether we see that the number of elementsg considered is at mostn

y5000 .

Remark. As the proof shows (see also the proof of Proposition 17 and the preceding discussion)the conclusion of Proposition 18 holds under the much weakerassumption that all elements of order3 in G move at least100 000 points.

Proposition 19. Assume thatm ≥ 100 000, k ≤ n23+ 1

100 , y 6= 0 and n is sufficiently large.Then the number of permutationsg ∈ G with |supp(g)| = k and |supp(gA)| = y is at most(nk

) 12n− y

2+ y

5000 k!2(a+4)k .

Proof. By Corollary 15 the number of possibilities for supp(gS∪L) is at most(nk

) 12[y2

]!n− y

2 · 2(a+4)k. Therefore the number of possibilities forgS∪L is at most

(n

k

) 12

n− y2 2(a+4)k

[y2

]!(k − y)! ≤

(n

k

) 12

n− y2 2(a+4)k · k!.

Hence by Proposition 18 the number of choices forg is at most

(n

k

) 12

n− y2+ y

5000 · k!2(a+4)k

19

as required.

The next result as a counterpart of Lemma 16 deals with the case whenny is large compared tok! (andy is large compared tom).

Corollary 20. Assume thatk ≤ n23+ 1

100 , ny8 ≥ k! andm is sufficiently large. Then the number of

permutationsg with |supp(g)| = k andsupp(gA)| = y is at most(nk

) 12n− y

5 .

Proof. We havem ≤ k ≤ n, hence ifm is large enough Proposition 19 is applicable. Moreover,we have2(a+4)k ≤ k! if m is large enough (compared to the fixed constanta).

Hence in this case we have

(n

k

)12

n− y2+ y

5000(k!2(a+4)k

)≤(n

k

) 12

n− y2+ y

5000+ y

4 ≤(n

k

) 12

n− y5 .

To deal with the case whenk! andny are “almost equal” we have to introduce further ideas andnotation. We call a pair of the form(supp(gS), gL) thick if the elementsg which correspond to itact in at least(k!)

16 different ways on supp(gS) and call a pairthin otherwise.

Proposition 21. Assume thatm ≥ 100 000, 2200a ≤ k ≤ n23+ 1

100 , y 6= 0 andn is sufficientlylarge. Then the number of permutationsg with |supp(g)| = k and |supp(gA)| = y for which

(supp(gS), gL) is thin is at most(nk

) 12 (k!)

16+ 1

100 .

Proof. By Corollary 15 the number of possibilities for the pair(supp(gS), gL) is at most(nk

) 12[y2

]!n− y

2 ·2(a+4)k . Hence the number of possibilities forgS∪L is at most

(n

k

) 12 [y

2

]!n− y

2 · 2(a+4)k(k!)16 ≤

(n

k

) 12 [y

2

]!n− y

2 (k!)16+ 1

100

(we used the condition200a ≤ log k). Using Proposition 18 we see that the total number ofelementsg considered is at most

(n

k

) 12

(k!)16+ 1

100n− y2+ y

5000

[y2

]! ≤

(n

k

) 12

(k!)16+ 1

100

(usingy ≤ k ≤ n23+ 1

100 ).

Proposition 22. Let (supp(gS), gL) be a thick pair. Denote the action of (the stabiliser ofsupp(gS)in) G on supp(gS) by H. There is an elementγ which corresponds to this pair such that thecentraliser ofγS in H has order at most

(5a)k−x−y

[k − x− y

2

]!/(k!)

16 .

20

Proof. By Lemma 14(2)H has order at mostak−x−y[k−x−y

2

]!. By a result of Kovacs and Robin-

son [KR93] the numberk(H) of conjugacy classes of the permutation groupH is at most5k−x−y.Using a well-known identity we obtain

∑

h∈HCH(h) = k(H)|H| ≤ (5a)k−x−y

[k − x− y

2

]! .

Since by definition we have at least(k!)16 choices forgS ∈ H, at least one of them has small

centraliser as required.

Proposition 23. Assume thatm ≥ 100 000, k ≥ 2100 000 and y 6= 0. Let(supp(gS), gL) be a thick pair andγ a corresponding permutation with small centraliser as above.The number of elementsg which correspond to this pair and satisfy the condition

|supp(gA) ∩ supp(γA)| ≥y

100

is at mostakkk4+ k

10 000n0.4951y/ [y

2

]!.

Proof. The number of choices for the set supp(gA) ∩ supp(γA) is less than2y. The number ofchoices for the rest of supp(gA) is at most

(n

[0.99 y]

). Given these sets (and hence supp(g)) by

Theorem A the number of choices forg is at mostkk

10 000 . It follows that the number of choices forg is less than

2kn0.99ykk

10 000/ [y

2

]! .

Another estimate for the number of possible choices forg is the following. The number of choices

for gS∪L is at mostak−x−y[k−x−y

2

]! by Lemma 14(2). Hence by Proposition 18 the number of

choices forg is less than

ak−x−y

[k − x− y

2

]!n

y5000 ≤ akk

k2n

y5000

/ [y2

]! .

A third estimate follows immediately from these; the numberof choices forg is at most

(ak · k k

2ny

5000 · 2k · n0.99y · k k10 000

) 12/ [y

2

]! ≤ akk

k4+ k

10 000n0.4951y/ [y

2

]!

as required.

Proposition 24. Assume thatm ≥ 100 000 andk ≥ 2100 000. Let (supp(gS), gL) be a thick pairand γ a corresponding permutation with small centralizer (as in Proposition 22). The number ofelementsg which correspond to this pair and satisfy

|supp(gA) ∩ supp(γA)| ≤y

100

is at mostn

y30 · k k

3+ k

10 000 (5a)k/ [y

2

]! .

21

Proof. Let us consider the commutator[γ, g]. By [DM96, Exercise 1.6.7] we have

|supp([γ, g]) ∩A| ≤ 3|supp(gA) ∩ supp(γA)| ≤3y

100.

Hence the number of choices for supp([γ, g])∩A is at mostn3y100 . Note that supp([γ, g])∩(S∪L) ≤

supp(γS∪L) (which is fixed). Using Theorem A we obtain that the number of choices for[γ, g] is at

mostn3y100 · k k

10 000 . This commutator, together withγ, determinesg−1γg = γ[γ, g]. If h is anotherelement withh−1γh = g−1γg, thengh−1 centralisesγ. Hence by the choice ofγ in Proposition 22)the number of possibilities forhS is less than

(5a)k[k − y

2

]!/

(k!)16 .

Hence we have at mostn3y100 · k k

10 000 (5a)k[k−y2

]!/

(k!)16 choices forgS∪L and given this, the

number of choices forg is at mostny

5000 by Proposition 18. Therefore the number of choices forgis at most

n3y100 · n y

5000 (5a)kkk

10 000

[k

2

]!/

(k!)16

[y2

]! ≤ n

y30 · k k

3+ k

10 000 (5a)k/ [y

2

]!

Our next result which builds on most of the earlier ones in this section implies Theorem B ifyis large compared tom.

Lemma 25 (Main Lemma). Assume thatk ≤ n23+ 1

100 , y 6= 0 andm is sufficiently large. Then the

number of permutationsg with |supp(g)| = k and |supp(gA)| = y is at most(nk

) 12 (k!)

14n− y

200 .

Proof. By Lemma 16 and Corollary 20 we may assume thatn3y ≥ k! ≥ ny8 . By Proposition 21 the

number of permutationsg with a thin pair(supp(gS), gL) is at most(n

k

) 12

(k!)16+ 1

100 ≤(n

k

) 12

(k!)14 (k!)−

120 ≤

(n

k

)12

(k!)14n− y

160 .

It remains to bound the number of permutationsg with a thick pair. By Corollary 15 the number of

possibilities for(supp(gS), gL) is at most(nk

) 12[y2

]!n− y

2 · 2(a+4)k. Given this, by Propositions 23and 24 the number of choices forg is at most

(akk

k4+ k

10 000n0.4951y + (5a)kkk3+ k

10 000ny30) / [y

2

]!

≤ (10a)k(k!)14n

3y10 000

(n0.4951y + n

y4 · n y

30) / [y

2

]!

≤ (10a)k(k!)14n0.4954y

/ [y2

]!

(we used the inequalityn3y ≥(ke

)k). Hence the total number of permutationsg with a thick pair is

at most(nk

) 12 (k!)

14n−0.006y

((10a)k2(a+4)k

). If m and hencek is large enough, then

(10a)k2(a+4)k ≤ 1

2(k!)

13000 ≤ 1

2n

y1000 .

Our statement follows.

22

Next we prove Theorem B in the case whenx is large compared tom.

Proposition 26. Assume thatx 6= 0, n23+ 1

100 ≥ k ≥ 2100 000 andm is sufficiently large. Thenthe number of permutationsg with |supp(g)| = k, |supp(gL)| = x and |supp(gA)| = y is at most(nk

) 12 (k!)

14n− x

20 000 .

Proof. If y ≥ x100 , then our statement follows from the Main Lemma. Assume now thaty ≤ x

100 .

By Lemma 14 the number of choices for supp(g) is at most(

n[k−x−y

2 ]

)2aknx( 1

7+ 1

100) ·(ny

). Hence,

by Theorem A the number of choices forg is at most(

n[k−x−y

2 ]

)2aknx( 1

7+ 1

100)(ny

)k

k10 000 (since we

can assume thatm ≥ 100 000). Using Proposition 12 andy ≤ x100 we see that this is at most

(n

k

) 12

2(a+2)kkk

10 000nx( 17+ 1

100)(n

y

)/( n[x+y2

])

≤(n

k

) 12

2(a+2)kkk

10 000nx( 17+ 2

100 )/ ( n[

x2

]).

If m and hencek is large enough compared toa, we have2(a+2)kkk

10 000 ≤ (k!)14 . Using100 ≤

x ≤ k ≤ n23+ 1

100 we see thatnx( 17+ 2

100)/(

n[x2 ]

)≤ n− x

100 . Our statement follows.

Let us return to the notation introduced after Lemma 13. IfSi ∈ S is a small block, such thatg moves at least3 points ofSi, then we denote|supp(g) ∩ Si| by zi. We setz(g) =

∑zi (for all

suchi).

Proposition 27. Assume thatz 6= 0, n23+ 1

100 ≥ k ≥ 2100 000 andm is sufficiently large. Then the

number of permutationsg with z(g) = z is at most(nk

) 12 (k!)

14n− z

800 000 .

Proof. If x ≥ z40 or y ≥ z

80 , then our statement follows from Lemma 25 and Proposition 26.Assume otherwise. Ifg moves a point of some block, then it moves at least two points of the block.Hence the number of blocks inS which contain two points from supp(g) is at most

[k−z2

]. These

blocks can be chosen in at most(

n[k−z

2 ]

)ways. The blocks inS which contain at least3 points

from supp(g) can be chosen in at most(

n[ z3 ]

)ways. Given these blocks the number of choices for

supp(gS) is at most

(n[

k−z2

])(

n[z3

])2aza2[

k−z2

] ≤(n

k

) 12

22k · 2ak(

n[z3

]) / ( n⌈

z2

⌉).

Usingn23+ 1

100 ≥ k ≥ z we see that(

n[ z3 ]

)n

z20 ≤

(n⌈ z

2⌉)2z. Hence the number of choices for

supp(g) is at most (n

k

) 12

2(a+3)kn− z20nx+y ≤

(n

k

) 12

2(a+3)kn− z80 .

The number of choices forg itself is at most(nk

) 12n− z

80 2(a+3)kkk

10 000 which is less than(nk

) 12n− z

80 (k!)14

if k is large enough.

23

Denote the number of small blocksSi ∈ S fixed byg such that|supp(g) ∩ Si| = 2 by v(g). Onthese blocksg acts as a transposition.

Proposition 28. Assume thatn23+ 1

100 ≥ k ≥ 2100 000 andm is sufficiently large, then the number

of permutationsg with v(g) = v ≥ m10 is at most

(nk

) 12 (k!)

14n− m

800 000 000 .

Proof. If x+y+z ≥ m1000 , then our statement follows from the previous results. Assume otherwise.

Suppose first thatk! ≥ nm100 . The number of choices for small blocksSi with |supp(g) ∩ Si| = 2 is

at most(

n[k2 ]

). Hence the number of choices for all the pairs supp(g)∩Si in these blocks is at most

(n[k2 ]

)(a2)[

k2 ] ≤

(nk

) 12 (2a)k. The number of choices for supp(g) is then at most

(n

k

) 12

(2a)knx+y+z ≤(n

k

) 12

(2a)k(k!)110 .

Hence by Theorem A the number of choices forg itself is at most(nk

) 12 (2a)k(k!)

110 k

k10 000 which

is less then(nk

) 12 (k!)

18 if m is large enough. Therefore in this case the number of permutations g

is at most(nk

) 12 (k!)

14n− m

800 . Suppose now thatk! ≤ nm100 . To any permutationg ∈ G we assign

a permutationg obtained by “forgetting about”[m10

]transpositions in the small blocksSj of the

smallest indexj (which g fixes and for which|supp(g) ∩ Sj| = 2). Note that ifg = h then|supp(gh−1)| ≤ m

2 , hence we haveg = h. That isg uniquely determinesg. The number of choicesfor supp(g) is at most

(n[

k2

]−[m10

])aknx+y+z ≤

(n

k

) 12

(2a)kn−m10n

m1000 .

The number of choices forg and henceg is at most

(n

k

) 12

(2a)kn−m10n

m1000 · k! ≤

(n

k

) 12

n−m10n

m1000 (k!)2

if m is large enough. Hence in this case the number of choices forg is at most

(n

k

) 12

n−m10n

m1000n

m50 ≤

(n

k

)12

· n−m20 .

This completes the proof of the proposition.

We need the following auxiliary result.

Lemma 29. LetH be a permutation group of degreen such that each element of order3 moves atleast100 000 points. Assume thatk ≤ n

23 andk is sufficiently large. Then

|Hk| ≤(n

k

)12

(k!)2.

24

Proof. Let g ∈ H be a permutation with|supp(g)| = k and|supp(gA)| = y. Using Corollary 15 wesee that the number of choices for gS∪L is at most(nk

) 12[y2

]!2(a+4)kn− y

2 k! which is less than1k(nk

) 12 (k!)2n− y

2 if k is large enough. By the remark after

Proposition 18 the number of possibilities forg is at most1k(nk

) 12 (k!)2n− y

2 · n y5000 ≤ 1

k

(nk

) 12 (k!)2.

Summing over thek ways to chosey, our statement follows.

Proposition 30. Assume thatn23 ≥ k ≥ 210 000 andm is sufficiently large. Then the number of

permutationsg with v(g) = v ≤ m10 is at most

(nk

) 12 (k!)

14n− m

800 000 000 .

Proof. Just like in the proof of Proposition 28 we might assume thatx + y + z ≤ m1000 . Note

that in the proof of Proposition 28 we do not use the conditionon v in the casek! ≥ nm100 , so

our statement follows in this case. Now assume thatk! ≤ nm100 . The number of choices for the

x + y + z + 2v points of supp(g) which are not contained in the two-element blocks moved bygis at mostnx+y+z+2v ≤ n

m5+ m

1000 . Let us fix such a setR of x + y + z + 2v points and count thepermutationsg which correspond toR. Denote byP the set of two-element blocks disjoint fromR.Each of the permutationsg considered induces a permutationg of P of support12(k − |R|). It is

clear that supp(g) andR determine supp(g). Assume first thatk ≥ |P| 23 . In this case the number

of choices for the two-element blocks moved byg is at most|P|k2 ≤ k34k ≤ k!. Hence the number

of choices for supp(g) is at mostnm4 · k! ≤ n

m4+ m

100 . Applying Theorem A, the number of choices

for g itself is bounded bynm4+ m

100 kk

10 000 ≤(nk

) 12 . In this case our statement follows. Assume now

thatk ≤ |P| 23 . Consider the permutation groupG generated by all the permutationsg. We claimthat each element of order3 in G moves at leastm4 points (ofP). For otherwise leth be an element

of order3 in G with |supp(h)| ≤ m4 . Now h can be written as a producth = g1 . . . gt in G (where

the gi are from the above generating set ofG, i.e. eachgi comes from one of theg). Considerh = g1 . . . gt ∈ G. It has order divisible by3 and henceh2 is non-trivial. Buth2 moves onlypoints inR and the points corresponding to the two-element blocks in supp(h). Hence we have|supp(h2)| ≤ m

2 + |R| < m, a contradiction. Applying Lemma 29, we see that the number of

possibilities for supp(g) is at most

(|P|[k2 ]

) 12

(k!)2 ≤(

n[ k2 ]

) 12n

m50 if m is large enough. Hence the

number of choices for supp(g) is at most

(n[k2

]) 1

2

nm5+m

50+ m

100 ≤(

n[k2

]) 1

2

nm4 .

The number of choices forg is at most(

n[k2 ]

) 12n

m4 k! ≤

(n[k2 ]

) 12n

m4+ m

100 which implies our state-

ment.

Putting together Lemma 13, Proposition 28 and Proposition 30 we obtain Theorem B.

25

Acknowledgment

JK acknowledges support by the European Commission under the Integrated Projects RESQ, IST-2001-37559 and Qubit Applications (QAP) funded by the IST directorate as Contract Number015848, and by ACI Securite Informatique SI/03 511 and ANRAlgoQP grants of the French Re-search Ministry. AS wishes acknowledge an Israeli Science Foundation grant and a Bi-NationalScience Foundation United States - Israel grant.

References

[Bab81] L. Babai. On the order of uniprimitive permutation groups.Annals of Math., 113:553–568, 1981.

[Bab82] L. Babai. On the order of doubly transitive permutation groups.Invent. Math., 65:473–484, 1982.

[Bea97] R. Beals. Quantum computation of Fourier transforms over symmetric groups. InProc.29th STOC, pages 48–53, 1997.

[Cam81] P.J. Cameron. Finite permutation groups and finite simple groups.Bull. London Math.Soc., 13(1):1–22, 1981.

[DM96] J. D. Dixon and B. Mortimer.Permutation groups. Grad. Texts in Math., Springer, NewYork, 1996.

[EH99] M. Ettinger and P. Hoyer. On quantum algorithms for noncommutative hidden sub-groups.Adv. in Appl. Math., 25(3):239–251, 2000.

[FIM+03] K. Friedl, G. Ivanyos, F. Magniez, M. Santha, and P. Sen. Hidden translation and orbitcoset in quantum computing. InProceedings of 35th ACM Symposium on Theory ofComputing (STOC), pages 1–9, 2003.

[GGL96] R.L. Graham, M. Grotschel, and L. Lovasz, Editors.Handbook of Combinatorics, MITPress, 1996.

[G70] V.D.Goppa. A new class of linear error-correcting codes. Problems of InformationTransmission, 6:207-212, 1970.

[GSVV01] M. Grigni, L. Schulman, M. Vazirani, and U. Vazirani. Quantum mechanical algorithmsfor the nonabelian hidden subgroup problem. InProceedings of 33th ACM Symposiumon Theory of Computing (STOC), pages 68–74, 2001.

[GM98] R. Guralnick and K. Magaard. On the minimal degree of aprimitive permutation group.J. Algebra207:127–145, 1998.

[HRT00] S. Hallgren, A. Russell, and A. Ta-Shma. Normal subgroup reconstruction and quantumcomputation using group representations. InProceedings of 32th ACM Symposium onTheory of Computing (STOC), pages 627–635, 2000.

26

[H+06] S. Hallgren, C. Moore, M. Roetteler, A. Russell, P. Sen. Limits of Quantum CosetStates for Graph Isomorphism. InProceedings of 38th ACM Symposium on Theory ofComputing (STOC), pages 604–17, 2006.

[IMS01] G. Ivanyos, F. Magniez, and M. Santha. Efficient quantum algorithms for some in-stances of the non-abelian hidden subgroup problem. InProceedings of 13th ACMSymposium on Parallelism in Algorithms and Architectures, pages 263–270, 2001.

[Jor73] C. Jordan. Sur la limite de transitivite des groupesnon alternes. Bull. Soc. Math.France, 1:40–71, 1873.

[Jor75] C. Jordan. Sur la limite de degre des groupes primitifs qui contiennent une substitutiondonnee.J. reine angew Math., 79:248–258, 1875.

[KS05] J. Kempe and A. Shalev. The hidden subgroup problem and permutation group theory.Proceedings of the 16th ACM-SIAM Symposium on Discrete Algorithms (SODA), pages1118-1125, 2005.

[KR93] L. G. Kovacs and G. R. Robinson, On the number of conjugacy classes of a finite group,J. Algebra, 160:441–460, 1993.

[Kup03] G. Kuperberg. A subexponential-time algorithm forthe dihedral hidden subgroup prob-lem. SIAM J. Comput., 35(1):170-188, 2005.

[L82] M. W. Liebeck. Bounds for the orders of some transitivepermutation groups,Bull.LMS, 14:337–344, 1982.

[L84] M. W. Liebeck. On the orders of transitive permutationgroups,Bull. LMS, 16:523–524,1984.

[LS91] M. W. Liebeck and J. Saxl. Minimal degrees of primitive permutation groups, with anapplication to monodromy groups of covers of Riemann surfaces,Proc. LMS, 63:266–314, 1991.

[LSh01] M. Liebeck and A. Shalev. Diameters of finite simple groups: sharp bounds and appli-cations.Ann. of Math. (2), 154(2):383–406, 2001.

[LL97] F. Levy-dit-Vehel and S.Litsyn. Parameters of Goppacodes revisited.IEEE Transac-tions on Information Theory, 43(6):1811–1819, 1997.

[MS77] F. J. MacWilliams and N.J.A. Sloane.The theory of error-correcting codes. North-Holland, 1977

[MRRS04] C. Moore, D. Rockmore, A. Russell, and L. Schulman.The value of basis selection inFourier sampling: hidden subgroup problems for affine groups. In Proceedings of theACM-SIAM Symposium on Discrete Algorithms (SODA), 2004.

[MRS05] C. Moore, A. Russell, and L. Schulman. The symmetricgroup defies strong Fouriersampling. InProc. 46th Annual IEEE Symp. on Foundations of Computer Science(FOCS), pages 479-490, 2005.

27

[NC00] M.A. Nielsen and I.L. Chuang.Quantum Computation and Quantum Information.Cambridge University Press, Cambridge, UK, 2000.

[PS80] C. E. Praeger and J. Saxl. On the orders of primitive permutation groups,Bull. LMS,12:303–307, 1980.

[RB98] M. Rotteler and T. Beth. Polynomial-time solution to the hidden subgroup problemfor a class of non-Abelian groups. Technical report, Quantum Physics e-Print archive,1998.http://xxx.lanl.gov/abs/quant-ph/9812070.

[Reg04] O. Regev. A subexponential time algorithm for the dihedral hidden subgroup prob-lem with polynomial space. Technical report, Quantum Physics e-Print archive, 2004.http://xxx.lanl.gov/abs/quant-ph/0406151.

[Sag01] Bruce E. Sagan.The symmetric group, volume 203 ofGraduate Texts in Mathemat-ics. Springer-Verlag, New York, second edition, 2001. Representations, combinatorialalgorithms, and symmetric functions.

[Sho94] P.W. Shor. Algorithms for quantum computation: Discrete log and factoring. InProceedings of the 35th Annual Symposium on the Foundationsof Computer Science(FOCS), pages 124–134, Los Alamitos, CA, 1994. IEEE Computer Society.

28