Communicated by Graeme Mitchison

Sorting with Self-organizing Maps

Marco Budinich Dipartimento di Fisica and INFN, Via Valerio 2, 34127 Trieste, Italy

A self-organizing feature map Won der Malsburg 1973; Kohonen 1984) sorts n real numbers in O(n) time apparently violating the O(n1ogn) bound. Detailed analysis shows that the net takes advantage of the uni- form distribution of the numbers and, in this case, sorting in O(n) is possible. There are, however, an exponentially small fraction of patho- logical distributions producing O(n’) sorting time. It is interesting to observe that standard learning produced a smart sorting algorithm.

Sorting n numbers requires at least (log, n!l comparisons and the “Heap- sort” algorithm makes at worst O(n log n ) steps. ”Quicksort,” faster on average, slows to O(n2) in the worst case (Knuth 1981; Aho 1974).

The standard Kohonen algorithm can sort: the n numbers to be sorted represent the patterns and there is a string of n neurons. The neurons start from random weights and “adapt” to the distribution of the input patterns during learning.’ After learning each pattern is mapped onto a neuron and, scanning the neuron string, one gets a sorted list of the patterns.

Tests of a modified learning algorithm ( n I lo5) show that sorting time is fast and grows like O(n) apparently violating the [log, n!] bound.

Disentangling this curious result is not easy given the stochastic na- ture of the algorithm so I examined a deterministic procedure carefully derived from it. It is conceivable that the results apply also to the neural net.

The basic idea is to put the n numbers in a histogram with n bins and to scan it: a bin with one number gives immediately its final position; a bin with k numbers iterates the procedure. In summary:

1. find minimum (rn) and maximum ( M ) of the n numbers and save

2. put the n - 2 numbers in a histogram with n channels evenly dis- them;

tributed in [rn, MI;

’Erwin (1992), Ritter (1992), and Csabai (1992) contain exhaustive analysis of learn- ing; Budinich (1995) contains a simpler proof of ordering.

Neural Computation 7,1188-1190 (1995) @ 1995 Massachusetts Institute of Technology

Sorting with Self-organizing Maps 1189

CPU psec

This algorithm .. I.."... ........... I.." ..... I .............. ..... ..... ......

n 20000 40000 60000 80000 100000

Figure 1: Comparison of unitary sorting time for uniformly distributed num- bers.

3. scan the histogram taking the following actions depending on bin content k:

0 if k = 0 skip the bin;

0 if k = 1 append the content of the bin to the output list;

0 if k > 1 recursively call the procedure for k numbers.

A slightly refined version of this algorithm produces the plot showing the sorting time per pattern for n up to lo5 and for uniformly distributed numbers (Fig. 1). Sorting time is independent of n for this algorithm whereas it has the expected log n dependence for Heapsort.

Steps 1, 2, and 3 of the algorithm require cyn steps after which, at most, ,Bn patterns are still to be sorted: 0 5 ,B 5 ( n - 2 ) / n < 1 (the worst case has n - 2 patterns in one bin). The number of steps S ( n ) needed to sort is

S ( n ) = cyn + S(Pn) = cyn + a/?n + S(,B/,Bn)

with

n - 4 o < p t < - - < p < 1 n - 2

1190 Marco Budinich

taking S(x) = 0 for x < 1:

With the upper limit for /3 we obtain an O ( n 2 ) trend whereas any distribution of the patterns giving constant /j is sorted in O(n) . A com- binatorial argument (due to E. Milotti) shows that the fraction of distri- butions giving at least one bin containing k or more numbers decreases exponentially with k.

This explains how the network algorithm sorts so quickly and it is amazing that learning of a self-organizing net adapts so well to the properties of the pattern distribution. One could argue that it is not so infrequent that nature has clever tricks to solve particular instances of problems that in general are difficult.

References

Aho, A. V., Hopcroft, J. E., and Ullmann, J. D. 1974. The Design and Analysis of Computer Algorithms. Addison-Wesley, Reading, MA.

Budinich, M., and Taylor, J. G. 1995. On the ordering conditions for self- organizing maps. Neural Comp. 7(2), 284-289.

Csabai, I., Geszti, T., and Vattay, G. 1992. Criticality in the one-dimensional Kohonen neural map. Phys. Rev. A 46(10), R6181-R6184.

Erwin, E., Obermayer, K. and Schulten, K. 1992. Self organizing maps: Order- ing, convergence properties and energy functions. Biol. Cybernet. 67,47-55.

Knuth, D. E., 1981. The Ar t of Computer Programming-Volume I l l Sorting and Searching, 2nd ed. Addison-Wesley, Reading, MA.

Kohonen, T. 1984. Self-Organisation and Associative Memory. (3rd ed., 1989). Springer-Verlag, Berlin.

Ritter, H., Martinetz, T., and Schulten, K. 1992. Neural Computation and Self Organizing Maps: An Introduction, p. 303. Addison-Wesley, New York.

Von der Malsburg, Ch. 1973. Self-organising of orientation sensitive cells in striate cortex. Kybernetik 14, 85-100.

Received January 18,1994; accepted January 30, 1995.