Neurocognitive approach to Neurocognitive approach to computational creativitycomputational creativity

Neurocognitive approach to Neurocognitive approach to computational creativitycomputational creativity

Włodzisław Duch & CoWłodzisław Duch & Co

Department of Informatics, Nicolaus Copernicus UniversityDepartment of Informatics, Nicolaus Copernicus University, , ToruńToruń, Poland, Poland

DepDept.t. of Computer Science, School of Comp. Engineering of Computer Science, School of Comp. Engineering, ,

Nanyang Nanyang Technological University, SingaporeTechnological University, Singapore

International Symposium on Artificial Brain with Emotion and International Symposium on Artificial Brain with Emotion and Learning, August 24-25, Yonsei University, Seoul, KoreaLearning, August 24-25, Yonsei University, Seoul, Korea

PlanPlan

1. Few projects at NTU/NCU2. Most mysterious things about mind … 3. Creativity research: psychology and

neuroscience.4. Intuition and insight.5. Neurocognitive model of higher mental

functions.6. Words in the brain.7. Creation of novel words, puzzles and word

games. 8. Creativity research - perspectives.

Humanized InTerfaces (HIT)Humanized InTerfaces (HIT)Humanized InTerfaces (HIT)Humanized InTerfaces (HIT)

C2I = Center for Computational Intelligence, SCE NTU Flagship project, principal investigators: Wlodzislaw Duch & Michel Pasquier + 15 other staff members …

HIT is a computer/phone interface that can interact in a natural way with the user, accept natural input in form of speech and sound commands; text commands; visual input, reading text (OCR), gesture/lips recognition

HIT should have a robust understanding of user intentions for selected applications. HIT should respond and behave in a natural way. Cognitive architectures in simulated talking head user can relate to, an android head, or a robotic pet.

Major goals of the HIT project: • develop modular extensible software/hardware platform for HITs; • test/develop cognitive architectures, create interactive word games,

information retrieval and other applications on PCs;• extend HIT functionality adding new interactivity & behavior;• move it to portable devices (PDAs/phones) & broadband services.

Developmental Robot-Embedded Developmental Robot-Embedded Artificial Mind (DREAM)Artificial Mind (DREAM)

Developmental Robot-Embedded Developmental Robot-Embedded Artificial Mind (DREAM)Artificial Mind (DREAM)

Centre for Computational Intelligence, NTU, Singapore.

• Develop a robotic head endowed with complex cognitive processor that

recognizes and interacts with humans in natural way. • To facilitate integration of research areas in perception (signal

processing, computer vision), real-time control, natural language

processing and cognitive modeling; use best available modules.• To create new applications for educational games, office assistants,

chatterbot interfaces, etc. • Test different cognitive architectures in realistic real-time control

setting: ACT-R, SOAR, but also new ideas such as IDA, Shruti.

DREAM modules DREAM modules

Natural input

modules Cognitive functions

Affectivefunctions

Web/text/databases interface

Behavior control

Control of devices

Talking head

Text to speechNLP

functions

Specializedagents

DREAM project is focused on perception (visual, auditory, text inputs), cognitive functions (reasoning based on perceptions), natural language communication in well defined contexts, real time control of the simulated/physical head.

Humanized interface

Store

Applications, eg. 20 questions game

Query

Semantic memory

Parser

Part of speech tagger& phrase extractor

On line dictionaries

Manual

verification

Emovere: A Neuro-Cognitive Computational Emovere: A Neuro-Cognitive Computational Framework for Research on EmotionsFramework for Research on Emotions

NTU/NCU Co-PIs: D. Cho, Quek H.C, W. Duch, Ng G.S, A. Wahab + John Taylor (King’s College London) and Looi Chee Kit, (NIE/LSL)

• Essential for HIT, DREAM and intelligent tutor projects that will benefit from affective computing.

• Emotions are an important factor in intelligent behavior, including problem solving, they can help to focus attention on correct reasoning.

• Understanding on how to capture real emotions in artificial system.

• Research on computational approaches to emotions is a state-of-the-art basic research topic.

IDoCareIDoCare: : IInfant nfant DDeveleveloopment and pment and CareCare for development of perfect babies!for development of perfect babies!

W. Duch, D.L. Maskell, M.B. Pasquier, B. Schmidt, A. Wahab

School of Computer Engineering, Nanyang Technological University

Problem: about 5-10% of all children have a developmental disability that causes problems in their speech and language development.

Identification of congenital hearing loss in USA is at 2½ years of age!

Solution: permanent monitoring of babies in the crib, stimulation, recording and analysis of their responses, providing guideline for their perceptual and cognitive development, calling an expert help if needed.

Key sensors: suction response (basic method in developmental psychology), motion detectors, auditory and visual monitoring.

Potential: market for baby monitors (Sony, BT...) is billions of $; so far they only let parents to hear or see the baby and play ambient music.

Other projectsOther projectsOther projectsOther projects

Brain as Complex System (BRACS) global brain simulation (EU project just submitted, coordinated by J.G. Taylor, KCL, UK).

Goal: autonomous robots, understanding how higher-level cognitive functions arise from neural interactions in a simplified brain architecture.

Failed in the last EU FP6 call, but will be updated for FP7 ...

Brain-Inspired Model of Skill Learning: From Conscious Cognition to Subconscious Actions (submitted to ARC).

Goal: using brain information flow simulations understanding the dynamics of the skill learning on the example of car driving.

Brain stem models, especially respiratory functions ...

Machine learning: meta-learning, learning all Boolean functions ...

Always looking for partners and funds !

Most mysterious Most mysterious … … What features of our brain/minds are most mysterious?

Consciousness? Imagination? Intuition? Emotions, feelings? Higher mental functions?

Masao Ito (director of RIKEN, neuroscientist) answered: creativity. Lady Lovelace (Turing 1950) wrote about Babbage analytical engine: „It cannot originate anything, it merely does that which we order it to do”.

MIT Encyclopedia of Cognitive Sciences has 1100 pages.

It has 6 chapters about logics & over 100 references to logics in the index. Creativity: 1 page (+1 page about „creative person”). Intuition: 0, not even mentioned in the index.

In everyday life we use intuition more often than logics.

How to define creativity?How to define creativity?Bink & Marsh (2001): the number of definitions of „creativity” is equal to the number of researchers that study this subject.

Sternberg (ed. Handbook of Human Creativity, 1998): „the capacity to create a solution that is both novel and appropriate”, not only in creation of novel theories or inventions, but also in our everyday actions, language understanding, interactions. Encyclopedia of creativity (Elsevier, 2005), eds. M. Runco & S. Pritzke, 167 articles, but no testable models of creativity have been proposed.

Journals: Creativity Research Journal, from 1988, LEA.Journal of Creative Behavior, from 1967, Creative Education Foundation.

Many connections with research in:

general intelligence, IQ tests, genius, special gifts, idiot savant syndrome and psychopathologies, intuition, insight (Eureka or Aha!), discovery ...

Psychology of Psychology of creativitycreativity

Psychology of Psychology of creativitycreativity

G. Wallas, The art of thought (1926): four-stage Gestalt model of problem solving. 4 stages: preparation, incubation, illumination and verification.

These stages were identified in creative problem solving by individuals and small groups of people; additional stages may involve: preparation stage preceded by finding or noticing a problem, proposing interesting questions, frustration period preceding illumination, final stage of communication that follows the verification stage.

Understanding details of such stages and sequences yielding creative productions is a central issue for creativity research, but is it sufficient?

Poincare (1948): math intuition and creativity is a discrimination between promising and useless ideas and their combinations; math thinking may be based on heuristic search among sufficiently rich representations. Math intuition is an interplay between spatial imagination, abstraction and approximate reasoning, and analytical reasoning or visual-spatial and linguistic thinking, observed in fMRI imaging (S. Dehaene, 1999).

IntuitionIntuitionIntuitionIntuitionIntuition is also a concept difficult to grasp, but commonly believed to play important role in business and other decision making; „knowing without being able to explain how we know”.

Sinclair & Ashkanasy (2005): intuition is a „non-sequential information-processing mode, which comprises both cognitive and affective elements and results in direct knowing without any use of conscious reasoning”.

First tests of intuition were introduced by Wescott (1961), now 3 tests are used, Rational-Experiential Inventory (REI), Myers-Briggs Type Inventory (MBTI) and Accumulated Clues Task (ACT).

Different intuition measures are not correlated, showing problems in constructing theoretical concept of intuition. Significant correlations were found between REI intuition scale and some measures of creativity. Intuition may result from implicit learning of complex similarity-based evaluation that are difficult to express in symbolic (logical) way.

Intuition in chess has been studied in details.

Insights and brainsInsights and brainsInsights and brainsInsights and brainsActivity of the brain while solving problems that required insight and that could be solved in schematic, sequential way has been investigated. E.M. Bowden, M. Jung-Beeman, J. Fleck, J. Kounios, „New approaches to demystifying insight”. Trends in Cognitive Science 2005.

After solving a problem presented in a verbal way subjects indicated themselves whether they had an insight or not.

An increased activity of the right hemisphere anterior superior temporal gyrus (RH-aSTG) was observed during initial solving efforts and insights. About 300 ms before insight a burst of gamma activity was observed, interpreted by the authors as „making connections across distantly related information during comprehension ... that allow them to see connections that previously eluded them”.

Insight interpretedInsight interpretedInsight interpretedInsight interpreted

What really happens? My interpretation:

• LH-STG represents concepts, S=Start, F=final• understanding, solving = transition, step by step, from S to F• if no connection (transition) is found this leads to an impasse; • RH-STG ‘sees’ LH activity on meta-level, clustering concepts into

abstract categories (cosets, or constrained sets);• connection between S to F is found in RH, leading to a feeling of

vague understanding; • gamma burst increases the activity of LH representations for S, F

and intermediate configurations; • stepwise transition between S and F is found;• finding solution is rewarded by emotions during Aha! experience;

they are necessary to increase plasticity and create permanent links.

Memory & creativityMemory & creativityMemory & creativityMemory & creativityCreative brains accept more incoming stimuli from the surrounding environment (Carson 2003), with low levels of latent inhibition responsible for filtering stimuli that were irrelevant in the past. “Zen mind, beginners mind” (S. Suzuki) – learn to avoid habituation! Complex representation of objects and situations kept in creative minds.

Pair-wise word association technique may be used to probe if a connection between different configurations representing concepts in the brain exists.

A. Gruszka, E. Nęcka, Creativity Research Journal, 2002.

Words may be close (easy) or distant (difficult) to connect; priming words may be helpful or neutral; helpful words are either semantic or phonological (hogse for horse); neutral words may be nonsensical or just not related to the presented pair.

Results for groups of people who are less/highly creative are surprising …

Word 1 Priming 0,2 s Word 2

Creativity & associationsCreativity & associationsCreativity & associationsCreativity & associationsHypothesis: creativity depends on the associative memory, ability to connect distant concepts together. Results: creativity is correlated with greater ability to associate words & susceptibility to priming, distal associations show longer latencies before decision is made.

Neutral priming is strange!

• for close words and nonsensical priming words creative people do worse than less creative; in all other cases they do better.

• for distant words priming always increases the ability to find association, the effect is strongest for creative people. Latency times follow this strange patterns.

Conclusions of the authors:

More synaptic connections => better associations => higher creativity.

Results for neutral priming are puzzling!

Words in the brainWords in the brainWords in the brainWords in the brainThe cell assembly model of language has strong experimental support; F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press.

Acoustic signal => phonemes => words => semantic concepts.Semantic activations are seen 90 ms after phonological in N200 ERPs.

Phonological density of words = # words that sound similar to a given word, that is create similar activations in phonological areas.

Semantic density of words = # words that have similar meaning, or similar extended activation network.

Perception/action networks, results from ERP & fMRI.

Words: simple modelWords: simple modelWords: simple modelWords: simple modelGoals: • make the simplest testable model of creativity; • create interesting novel words that capture some features of products;• understand new words that cannot be found in the dictionary.

Model inspired by the putative brain processes when new words are being invented. Start from keywords priming auditory cortex.

Phonemes (allophones) are resonances, ordered activation of phonemes will activate both known words as well as their combinations; context + inhibition in the winner-takes-most leaves only a few candidate words.

Creativity = imagination (fluctuations) + filtering (competition)

Imagination: chains of phonemes activate both word and non-word representations, depending on the strength of the synaptic connections. Filtering: based on associations, emotions, phonological/semantic density.

Paired associationsPaired associationsPaired associationsPaired associationsSo why neutral priming for close associations and nonsensical priming words degrades results of creative people?

High creativity = many connections between microcircuits; nonsensical words add noise, increasing activity between many circuits; in a densely connected network adding noise creates confusion, the time need for decision is increased because the system has to settle in specific attractor.

If creativity is low and associations distant, noise does not help, because there are no connections, and priming words contribute only to chaos. Nonsensical words increase overall activity in the intermediate configura-tions. For creative people resonance between distant microcircuits is possible: this is called stochastic resonance, observed in perception.

For priming words with similar spelling and close words the activity of the second word representation is higher, always increasing the chance of connections and decreasing latency. For distant words it will not help, as intermediate configurations are not activated.

Words: algorithmWords: algorithmWords: algorithmWords: algorithmNeural resonant models (~ ARTWORD), or associative nets.Simplest things first => statistical model.

Preliminary: • create probability models for linking phonemes and syllables;• create semantic and phonological distance measures for words.

Statistical algorithm to find novel words:

• Read initial pool of keywords.

• Find phonological and semantic associations to increase the pool.

• Break all words into chains of phonemes, and chains of morphemes.

• Find all combinations of fragments forming longer chunks ranked according to their phonological probability (using bi- or tri-grams).

• For final ranking use estimation of semantic density around morphemes in the newly created words.

Words: experimentsWords: experimentsWords: experimentsWords: experimentsA real letter from a friend: I am looking for a word that would capture the following qualities: portal to new worlds of imagination and creativity, a place where visitors embark on a journey discovering their inner selves, awakening the Peter Pan within. A place where we can travel through time and space (from the origin to the future and back), so, its about time, about space, infinite possibilities.FAST!!! I need it sooooooooooooooooooooooon.

creativital, creatival (creativity, portal), used in creatival.comcreativery (creativity, discovery), creativery.com (strategy+creativity)discoverity = {disc, disco, discover, verity} (discovery, creativity, verity)digventure ={dig, digital, venture, adventure} still new! imativity (imagination, creativity); infinitime (infinitive, time) infinition (infinitive, imagination), already a company namejournativity (journey, creativity) learnativity (taken, see http://www.learnativity.com)portravel (portal, travel); sportal (space, sport, portal), taken timagination (time, imagination); timativity (time, creativity)tivery (time, discovery); trime (travel, time)

Word gamesWord gamesWord gamesWord gamesWord games were popular before computer games. They are essential to the development of analytical thinking. Until recently computers could not play such games.

The 20 question game may be the next great challenge for AI, because it is more realistic than the unrestricted Turing test; a World Championship with human and software players (in Singapore)?

Finding most informative questions requires knowledge and creativity.

Performance of various models of semantic memory and episodic memory may be tested in this game in a realistic, difficult application.

Asking questions to understand precisely what the user has in mind is critical for search engines and many other applications.

Creating large-scale semantic memory is a great challenge: ontologies, dictionaries (Wordnet), encyclopedias, MindNet (Microsoft), collaborative projects like Concept Net (MIT) …

Puzzle generatorPuzzle generatorPuzzle generatorPuzzle generator

Semantic memory may be used to invent automatically a large number of word puzzles that the avatar presents. This application selects a random concept from all concepts in the memory and searches for a minimal set of features necessary to uniquely define it; if many subsets are sufficient for unique definition one of them is selected randomly.

It has charm, it has spin, and it has charge. What is it?

It is an Amphibian, it is orange and has black spots. How do you call this animal?

A Salamander.

If you do not know, ask Google!Quark page comes at the top …

Text understandingText understandingText understandingText understanding

Neurocognitive approach to language understanding: use recognition, semantic and episodic memory models, create graphs of consistent concepts for interpretation, use spreading activation and inhibition to simulate effect of semantic priming, annotate and disambiguate text.

For medical texts ULMS has >2M concepts, 15M relations … created a System for Unambiguous Concept Mapping in Medical Domain (with Matykiewicz, Pestian), and ontology for common reason (with Szymanski)

Graphs of consistent conceptsGraphs of consistent conceptsGraphs of consistent conceptsGraphs of consistent conceptsGeneral idea: when the text is read and analyzed activation of semantic subnetwork is spread; new words automatically assume meanings that increases overall activation, or the consistency of interpretation.Many variants, all depend on quality of semantic network, some include explicit competition among network nodes.1. Recognition of concepts associated with a given concept:

1.1 look at collocations, and close co-occurrences, sort using average distance and # occurrences;

1.2 accept if this is a ULMS concept; manually verify if not;

1.3 determine fine semantic types, what states/adjectives can be applied.

2. Create semantic network:

2.1 link all concepts, determine initial connection weights (non-symmetric);

2.2 add states/possible adjectives to each node (yes/no/confirmed …).

GCC analysisGCC analysisGCC analysisGCC analysisAfter recognition of concepts and creation of semantic network:3. Analyze text, create active subnetwork (episodic working memory) to make inferences, disambiguate, and interpret the text.

3.1 find main unambiguous concepts, activate and spread their activations within semantic network; all linked concepts become partially active, depending on connection weights.

3.2 Polysemous words, acronyms/abbreviations in expanded form, add to the overall activation; active subnetwork activates appropriate meanings stronger than other meaning, inhibition between competing interpretations decreases alternative meanings.

3.3 Use grammatical parsing and hierarchical semantic types constraints (Optimality Theory) to infer the state of the concepts.

3.4 Leave only nodes with activity above some threshold (activity decay).

4. Associate combinations of particular activations with billing codes etc.

Commercial breakCommercial breakCommercial breakCommercial breakIs creativity based on unconstrained imagination, no rules?

No! Anarchist type of methods encouraging unstructured approach fail (including free associations, brainstorming, random stimulation or lateral thinking)! Structured approaches, based on higher-order rules and templates, lead to excellent results; see:

Goldenberg, Mazursky & Solomon, Science 285, 1999.J. Goldenberg & D. Mazursky, Creativity in Product Innovation, CUP 2002

270 possible traits (T) were collected from adds in magazines; 900 symbols (S) that people associated with these traits were collected.3-4 most frequent symbols were finally selected for each trait.Replacement schema for advertising of product P:

1. Define the relevant trait T for a given product P. 2. List symbols S that completely and unquestionably invoke T. 3. Construct P-space of objects that are strongly correlated with P.

4. Substitute an aspect A of one of the objects in place of the corresponding aspect of S.

Replacement schemeReplacement schemeReplacement schemeReplacement schemeTask: create advertisement for Nike air shoes.

Product P = Nike air shoes

Trait T: “cushioning and absorbing the shocks” caused by jumping.

Symbol S that invokes T: life net for fire victims jumping from a burning building.

Replace S with P.

Proposed advertisement:

firemen holding a giant shoe!Ideas generated by the automated routine were presented to judges, along with ideas on the same theme appearing in magazine ads and advertising ideas generated by layman individuals.

Magazine ads: 2.880.55, templates 2.890.48, laymens 2.220.43 Winning adds: 3.26 0.49

More addsMore addsMore addsMore adds

• An Apple Computer terminal offering flowers (for advertising Apple Computers friendliness).

• Temple Mountain Mosque with Tennis ball texture (for advertising World Cup Tennis Tournament in Jerusalem).

• A cuckoo in the shape of a plane emerging from the cuckoo clock (for advertising the time accuracy of an airline company).

• Two Jeeps communicating in sign language (for silent car engine).

• A bullet shaped car (for fast car).

Conclusion: creative machines are possible in many applications!

Future plansFuture plansFuture plansFuture plansDetailed neural model of creating novel words; comparison of human and machine-generated results; this is a well defined domain, experimental research using EEG, ERP, fMRI and other techniques are possible. Understanding of real reasons for decision making. Imagination and filtering – creativity in other domains, although conceptual structures, knowledge representation and filtering are harder to implement.

Creation of semantic memory from Internet and collaborative sources.World championships in word games?

Applications of neurocognitive approach to NLP:

• creation of novel brand names;

• creativity templates in various domains;

• word games, puzzles, educational tests in computers and phones;

• hope to reach human level competence in language understanding.

Thank Thank youyoufor for

lending lending your your ears ears

......

Google: Duch => Papers