Digits that are notGenerating new types through deep neural nets

Mehdi Cherti, Balázs Kégl LAL/LRI, CNRS

Université Paris-Saclay

Akın Kazakçı MINES ParisTech,

PSL Research University

The Short Story

Why are we doing this?

How to determine the value of novelty?

The value of novelty is the blindspot of creativity research (Kazakçı, 2014).

Computational creativityDeep learning to the rescue?

• has enabled great progress in machine learning

• Also, several promising work renewed interest in computational creativity.

• Google created Magenta (nobody needs us anymore)

Gatys et al. 2015

Deep learning

Yet• main emphasis in DL research remains

learning to predict• models are based on likelihood whereas

creativity is unlikely• the initial breakthrough came from a

generative model

But the generative potential of deep nets is largely unexplored

Fitness function barrier

• For most computational creativity systems, the value function is fixed and predetermined

• This is a paradox. - And an obstacle for progress.

• Evolutionary approaches to computational creativity bears these inherent limitations:

• Explicit fitness functions reflects system designer’s preference for novelty - not the machine’s.

• We call this the fitness function barrier.

A program to go beyond the barrier

Ultimate objective: Try & get rid of hard-coded value functions; let the system develop its own

A genuinely creative system needs to be able to develop its own notion of value

Current work: Build a system

1. that can study & learn a referential set of objects (RS)

2. that can generate new objects that keep essential features of RS but generate unseen types

3. that provides an experimental bench for developing & testing various ways an agent can develop a value function

Auto-associative neural netsa.k.a auto-encoders

Learning to disassemble

Learning to build

- Auto-encoders have existed for long time (Kramer 1991)

- Deep variants are more recent (Hinton, Salakhutdinov, 2006; Bengio 2009)

- A deep auto-encoder learns successive transformations that decompose and then recompose a set of training objects

- The depth allows learning a hierarchy of transformations

The experimental setup

- Training data : MNIST, 70000 images of handwritten digits of size 28x28

- We use a sparse convolutional auto-encoder (3c1d) trained to:

- Encode : take an image and transform it to a sparse code

- Decode : take the sparse code and reconstruct the image

- Training objective is to minimize the reconstruction error

Generating new symbols

- We use an iterative method to build symbols the has never seen:

- Start with a random image x0 = r,

- and force the network to construct (i.e. interpret)

- xk = f(xk-1), until convergence

- Our method is inspired by Bengio et al. (2013)

- By contrast to them, we do not constrain the net to generate only known types (we do not consider unknown symbols as spurious)

Visualising the structure of generated images

Coloured clusters are original MNIST digits (classes from 0 to 9)

The gray dots are newly generated objects

New objects form new clusters

Using a clustering algorithm, we recover coherent sets of new symbols

A distance preserving projection of digits to a two-dimensional space (van der Maaten and Hinton 2008)

Creativity by fixations

Lear

nFo

rce

Fixa

te

Summary

A genuinely creative system needs to be able to develop its own notion of value.

Our system is a first step:• It can effectively create new types of objects preserving abstract

and semantic properties of a domain. • It provides an experimental setup that enables testing various

hypothesis.• It provides a bridge between current research on machine

learning and creativity research.

Thank you

Mehdi Cherti, Balázs Kégl {mehdi.cherti, balazskegl}@gmail.com

Akın Kazakçı akin.kazakci@mines-paristech.fr