Matrix Factorizations over

Non-Conventional Algebras for

Data MiningPauli Miettinen 28 April 2015

Chapter 1. A Bit of Background

Data

✔ ✔ ✘

✔ ✔ ✔

✘ ✔ ✔

long-hairedwell-known

male

Data

long-hairedwell-known

male

1 1 01 1 10 1 1( )

Factorization point of view

1 1 01 1 10 1 1( )

1 0

1 1

0 1( ) 1 1 0

0 1 1( )= ×○

Chapter 2. Boolean Matrix Factorization

Gian-Carlo Rota Foreword to Boolean matrix

theory and applications by K. H. Kim, 1982

In the sleepy days when the provinces of France were still quietly provincial, matrices with Boolean entries were a favored occupation of aging professors at the universities of Bordeaux and Clermont-Ferrand. But one day…

“

Boolean products and factorizations

• The Boolean matrix product of two binary matrices A and B is their matrix product under the Boolean semi-ring

• The Boolean matrix factorization of a binary matrix A expresses it as a Boolean product of two binary factor matrices B and C, that is, A = B◦C

(A �B)�j =Wk

�=1 ��kbkj

Matrix ranks• The (Schein) rank of a matrix A is the least number of

rank-1 matrices whose sum is A

• A = R1 + R2 + … + Rk

• Matrix is rank-1 if it is an outer product of two vectors

• The Boolean rank of binary matrix A is the least number of binary rank-1 matrices whose element-wise or is A

• The least k such that A = B◦C with B having k columns

Comparison of ranks• Boolean rank can be less than normal rank

• rankB(A) = O(log2(rank(A))) for certain A

⇒ Boolean factorization can achieve less error than SVD

• Boolean rank is never more than the non-negative rank

0B@1 1 01 1 10 1 1

1CA

The many names of Boolean rank

• Minimum tiling (data mining)

• Rectangle covering number (communication complexity)

• Minimum bi-clique edge covering number (Garey & Johnson GT18)

• Minimum set basis (Garey & Johnson SP7)

• Optimum key generation (cryptography)

• Minimum set of roles (access control)

123

Boolean rank and bicliques

1 0

1 1

0 1( )1

2

3

A

B

C

1 1 0

1 1 1

0 1 1( )1

23

A B C

1 1 0

0 1 1( )o=

A B C

Boolean rank and sets• The Boolean rank of a

matrix A is the least number of subsets of U(A) needed to cover every set of the induced collection C(A)

• For every C in C(A), if S is the collection of subsets, have subcollection SC such that

1 3

2

SS2SC S = C

Approximate factorizations

• Noise usually makes real-world matrices (almost) full rank

• We want to find a good low-rank approximation

• The goodness is measured using the Hamming distance

• Given A and k, find B and C such that B has k columns and |A – B◦C| is minimized

• No easier than finding the Boolean rank

The many applications of Boolean factorizations

• Data mining

• noisy itemsets, community detection, role mining, …

• Machine learning

• multi-label classification, lifted inference

• Bioinformatics

• Screen technology

• VLSI design

• …

The bad news• Computing the Boolean rank is NP-hard

• Approximating it is (almost) as hard as Clique [Chalermsook et al. ’14]

• Minimizing the error is hard

• Even to additive factors [M. ’09]

• Given one factor matrix, finding the other is NP-hard

• Even to approximate well [M. ’08]

Some algorithms• Exact / Boolean rank

• reduction to clique [Ene et al. ’08]

• GreEss [Bělohlávek & Vychodil ’10]

• Approximate

• Asso [M. et al. ’06]

• Panda+ (error & MDL) [Lucchese et al. ’13]

• Nassau (MDL) [Karaev et al. ’15]

Chapter 3. Dioids Are Not Droids

Intuition of matrix multiplication

• Element (AB)ij is the inner product of row i of A and column j of B

�

�

�

Intuition of matrix multiplication

• Matrix AB is a sum of k matrices alblT

obtained by multiplying the l-th column of A with the l-th row of B

��

Remember at least this slide

• A matrix factorization presents the input matrix as a sum of rank-1 matrices

• A matrix factorization presents the input matrix as an aggregate of simple matrices

• What “aggregate” and “simple” mean depends on the algebra

Dioids are not droids• Dioid is also not a diode

• Dioid is an idempotent semiring S = (A, ⊕, ⊗, ⓪, ①)

• Addition ⊕ is idempotent

• a + a = a for all a ∈ A

• Addition is not invertible

Some examples (1)

• The Boolean algebra B = ({0,1}, ∨, ∧, 0, 1)

• The subset lattice L = (2U, ∪, ∩, ∅, U) is isomorphic to Bn

• The Boolean matrix factorization expresses matrix A as A ≈ B⊗BC where all matrices are Boolean

Some examples (2)

• Fuzzy logic F = ([0, 1], max, min, 0, 1)

• Generalizes (relaxes) Boolean algebra

• Exact k-decomposition under fuzzy logic implies exact k-decomposition under Boolean algebra

Fuzzy example0B@1 1 0 01 1 1 10 1 0 10 1 1 1

1CA ⇡

0B@1 01 10 10 1

1CA⌦FÅ1 1 0 10 1 2/3 1

ã

=

0B@1 1 0 01 1 2/3 10 1 2/3 10 1 2/3 1

1CA

Some examples (3)

• The or–Łukasiewicz algebra

• Ł = {[0,1], max, ⊗Ł, 0, 1}

• a ⊗Ł b = max(0, a + b – 1)

• Used to decompose matrices with ordinal values [Bělohlávek & Krmelova ’13]

Some examples (4)

• The max-times (or subtropical) algebra M = (ℝ≥0, max, ×, 0, 1)

• Isomorphic to the tropical algebra T = (ℝ∪{–∞}, max, +, –∞, 0)

• T = log(M) and M = exp(T)

Why max-times?

• One interpretation: Only strongest reason matters (a.k.a. the winner takes it all)

• Normal algebra: rating is a linear combination of movie’s features

• Max-times: rating is determined by the most-liked feature

Max-times example0B@1 1 0 01 1 1 10 1 0 10 1 1 1

1CA ⇡

0B@1 01 10 2/30 1

1CA⌦MÅ1 1 0 10 1 2/3 1

ã

=

0B@1 1 0 01 1 2/3 10 2/3 4/9 2/30 1 2/3 1

1CA

On max-times algebra

• Max-times algebra relaxes Boolean algebra (but not fuzzy logic)

• Rank-1 components are “normal”

• Easy to interpret?

• Not much studied

On tropical algebras• A.k.a. max-plus, extremal, maximal algebra

• Much more studied than max-times

• Can be used to solve max-times problems, but needs care with the errors

• If in max-plus then in max-times, where kX � eXk �kX0 �›X0k M2�M = exp(m�x�,j{X �j, eX �j})

More max-plus• Max-plus linear functions:

f(x) = fT⊗x = max{fi+xi}

• f(α⊗x ⊕ β⊗y) = α⊗f(x) ⊕ β⊗f(y)

• Max-plus eigenvectors and values: X⊗v = λ⊗v (maxj{xij + vj} = λ + vi for all i)

• Max-plus linear systems: A⊗x = b

• Solving in pseudo-P for integer A and b

Computationalcomplexity

• If exact k-factorization over semiring K implies exact k-factorization over B, then finding the K-rank of a matrix is NP-hard (even to approximate)

• Includes fuzzy, max-times, and tropical

• N.B. feasibility results in T often require finite matrices

Anti-negativity and sparsity

• A semiring is anti-negative if no non-zero element has additive inverse

• Some dioids are anti-negative, others not

• Anti-negative semirings yield sparse factorizations of sparse data

Chapter 4. Even More General

Community detection• Boolean factorization can be considered as a community detection

method

• But not all communities are cliques

• “Beyond the blocks”

• Are matrix factorizations outdated models for graph communities before they even took off?

Beyond Blocks: Hyperbolic Community Detection 51

0

100

200

300

400

500

600

0 100 200 300 400 500 600

Fig. 1. Motivation for our work: Realground-truth community

0

100

200

300

400

0 100 200 300 400

Fig. 2. Result of our work: Commu-nity found by HyCoM-FIT

degree distributions. As such, they are typically best represented as having anhyperbolic structure in the adjacency matrix, rather than rectangular (uniform)structure. We detail HyCoM - the Hyperbolic Community Model - as a bet-ter representation of communities and the relationships between their members,and introduce HyCoM-FIT as a scalable algorithm to detect communities withhyperbolic structure. To illustrate our model and algorithm, Figure 1 representsthe adjacency matrix of a real (ground-truth) community externally providedwhen nodes are ordered by degree, and Figure 2 shows the adjacency matrix ofan exemplary community found by our algorithm. Clearly, both communities donot show uniform density. In a nutshell, the main contributions of our work are:

– Introduction of the Hyperbolic Community Model: We provide empiri-cal evidence that communities in large, real social graphs are better modeledusing an hyperbolic model. We also show that this model is better from acompression perspective than previous models.

– Scalability: We develop HyCoM-FIT, an algorithm for the detection ofhyperbolic communities that scales linearly with the number of edges.

– No user-defined parameters: HyCoM-FIT detects communities in aparameter-free fashion, transparent to the end-user.

– Effectiveness: We applied HyCoM-FIT on real data where we discoveredcommunities that agree with intuition.

– Generality: HyCoM includes uniform block communities used by otheralgorithms as a special case.

2 Background and Related Work

Nodes in real-world networks organize into communities or clusters, which tendto exhibit a higher degree of ‘cohesiveness’ with respect to the underlying rela-tional patterns. Group formation is natural in social networks as people organizein families, clubs and political organizations; see e.g., [19]. Communities also

Generalized outer product

• A generalized outer product is a function o(x, y, θ)

• Returns an n-by-m matrix A

• If xi = 0 or yj = 0, then (A)ij = 0

• Compare to xyT

Example• Generalized outer product for biclique core

• Binary vector x to select the subgraph • Set C to define the nodes in the core

• (o(x, x, C))ij = 1 if xi = xj = 1 and exactly one of i and j is in C

0BB@

11...1

1CCA

�1 1 · · · 1�

} = C

Generalized decomposition

• A generalized matrix decomposition decomposes input matrix A into a sum of generalized outer products

• A = o(x1, y1, θ1) ⊕ o(x2, y2, θ2) ⊕ … ⊕ o(xk, yk, θk)

• Sum can be over any semi-ring

• The generalized rank is defined as expected

Why generalize?

• Provides an unifying framework

• Some algorithms and many computational hardness results generalize well

• Depend more on the addition ⊕ than on the outer product

Some results• Finding the largest-circumference rank-1 submatrix

is NP-hard if the outer product is hereditary

• Generalizes results for nestedness

• Given a set of binary rank-1 matrices, finding the smallest exact sub-decomposition from them is NP-hard if addition is either OR, AND, or XOR

• But exact hardness depends on the algebra

Chapter 5. The Chapter to Remember

Conclusions• Matrix factorizations are just a way to express

complex data as an aggregate of simple parts

• Normal and Boolean algebras are the best-studied ones

• but by no means the only possible ones

• Generalizing the outer product gives even more versatile language

Thank You!