ImageryImagery, Creativity, , Creativity, Brains and TalentBrains and Talent

ImageryImagery, Creativity, , Creativity, Brains and TalentBrains and Talent

Włodek Duch Włodek Duch Department of InformaticsDepartment of Informatics, ,

Nicolaus Copernicus UniversityNicolaus Copernicus University, Toru, Torun, Polandn, Poland. . Google: W. DuchGoogle: W. Duch

Hitachi Hitachi Advanced Research Laboratory, , July July 2020110 0

ToruToruńń

Norbert Tomek Marek KrzysztofNorbert Tomek Marek Krzysztof

CopernicuCopernicuss

Nicolaus Copernicus: born in Nicolaus Copernicus: born in Torun in Torun in 14721472

DI NCU ProjectsDI NCU Projects: CI: CIDI NCU ProjectsDI NCU Projects: CI: CI

Google W Duch => List of projects, talks, papers

Computational intelligence (CI), main themes: • Foundations of computational intelligence: transformation

based learning, k-separability, learning hard boole’an problems.• Meta-learning, or learning how to learn.• Understanding of data: prototype-based rules, visualization. • Novel learning: projection pursuit networks, QPC (Quality of

Projected Clusters), search-based neural training, transfer learning or learning from others (ULM), aRPM, SFM ...

• Similarity based framework for metalearning, heterogeneous systems, new transfer functions for neural networks.

• Feature selection, extraction, creation.

DI NCU ProjectsDI NCU Projects:NCI :NCI DI NCU ProjectsDI NCU Projects:NCI :NCI

Neurocognitive Informatics projects.

•Computational creativity, insight, intuition, consciousness.•Neurocognitive approach to language, word games. •Medical information retrieval, analysis, visualization. •Global analysis of EEG, visualization of high-D trajectories.•Brain stem models and consciousness in artificial systems.•Autism, comprehensive theory. •Imagery agnosia, especially imagery amusia.•Infants: observation, guided development. •A test-bed for integration of different Humanized Interface Technologies (HIT), with Singapore C2I Center. •Free will, neural determinism and social consequences.

Visual imageryVisual imageryVisual imageryVisual imageryLarge field, with several journals, ex:

• Journal of Mental Imagery (1977), official journal of the International Imagery Association (not much neuroscience).

• Imagination, Cognition and Personality (1980), mostly psychological. • Journal of Imagery Research in Sport and Physical Activity (2007).• Neuroimaging of Mental Imagery: A Special Issue of the European Journal of

Cognitive Psychology (2004)

S. V. Thompson, Visual Imagery: a discussion. Educ. Psych. 10, 1990 , 141-167

Individual differences in visual imagery, together with a lack of understanding that others may think in a radically different way in this respect, may have had a profound effect on theories of thought and knowledge, yet attempts to validate measures of this variable in terms of educationally significant correlates have been relatively unsuccessful.

• Verbalizers and Imagers, division important in education.Better tests? More subtle divisions? Statistics?

Imagery and brainsImagery and brainsImagery and brainsImagery and brainsHow and where are mental images formed?How and where are mental images formed?

• Borst, G., Kosslyn, S. M, Visual mental imagery and visual perception: Borst, G., Kosslyn, S. M, Visual mental imagery and visual perception: structural equivalence revealed by scanning processes. structural equivalence revealed by scanning processes. Memory & Cognition, 36, 849-862, 2008. Memory & Cognition, 36, 849-862, 2008.

The present findings support the claim that image representations The present findings support the claim that image representations depict information in the same way that visual representations do. depict information in the same way that visual representations do.

• Cui, X et al. (2007) Vividness of mental imagery: Individual variability can be Cui, X et al. (2007) Vividness of mental imagery: Individual variability can be measured objectively. Vision Research, 47, 474-478.measured objectively. Vision Research, 47, 474-478.

Reported Vividness of Visual Imagination (VVIQ) correlates well with the early Reported Vividness of Visual Imagination (VVIQ) correlates well with the early visual cortex activity relative to the whole brain activity measured by fMRI visual cortex activity relative to the whole brain activity measured by fMRI (r=-0.73), and the performance on a novel psychophysical task. (r=-0.73), and the performance on a novel psychophysical task. Findings emphasize the importance of examining individual subject Findings emphasize the importance of examining individual subject variability.variability.

Poor perceptual imagery: why? Weak top-down influences? Poor perceptual imagery: why? Weak top-down influences? Unable to draw from memory, describe details, faces, notice changes, etc. Unable to draw from memory, describe details, faces, notice changes, etc.

What is needed to have qualia?What is needed to have qualia?What is needed to have qualia?What is needed to have qualia?Sensory cortex, for example V4 for color, MT for movement.

Bottom-up and top-down activations create resonant states. What if top-down connections are weak or missing?

C. Gilbert, M. Sigman, Brain States: Top-Down Influences in Sensory Processing.Neuron, Volume 54, Issue 5, Pages 677-696, 2007

Cortical & thalamic sensory lower-level processes are shaped by complex top-down influences, attention, task-related expectations and sensory inputs. Brain states are determined by the interactions between multiple cortical areas and the modulation of intrinsic circuits by feedback connections. Disruption of this interaction may lead to behavioral disorders.

Dehaene et al, Conscious, preconscious, and subliminal processing. TCS 2006Bottom-up strength & top-down attention combined lead to 4 brain states, with both stimulus and attention required for conscious reportability. Imagery? Electric shock activates auditory cortex through amygdala pathway, same with imagined scary episodes.

Visual top-downVisual top-downVisual top-downVisual top-down• Normal perception requires top-down

influences to form expectations. • What if PC/FC feedback connections to

visual/auditory areas are weak?

• This does not qualify as agnosia, but is a kind of imagery agnosia, a syndrom that has not yet been clearly identified!

How will the weak top-down connections in visual modality manifest?

Attention problems? Only if signals are very weak (ex. in poor lighting conditions) object recognition may be impaired. Other symptoms: poor visual imagination, memory for visual features, inability to draw from memory, recall and describe faces and objects, notice changes, slow in making puzzles, difficulty to see 3D magic eye pictures, perhaps more introvert? More conceptual than perceptual thinking … recognition memory may work fine.

At PC/FC level less interferences from sensory areas, so imagination, creativity, reasoning are fine, perhaps even better than average.

Auditory PerceptionAuditory PerceptionAuditory PerceptionAuditory PerceptionMuch less research has been done with auditory perception than visual.

Peretz, I., Champod, S. & Hyde, K, Varieties of Musical Disorders: The Montreal Battery of Evaluation of Amusia. Annals of the New York Academy of Sciences, 999, 58-75, 2003. Model of music perception behind the MBAE test does not include imagery.

Music PerceptionMusic PerceptionMusic PerceptionMusic PerceptionCognitive model of music Cognitive model of music processing is focused on pitch and processing is focused on pitch and rhythm processing: pitch in lateral rhythm processing: pitch in lateral Heschl’s gyrus, timbre in posterior Heschl’s gyrus, timbre in posterior superior-temporal lobes, rhythm superior-temporal lobes, rhythm in motor/mesiolimbic areas.in motor/mesiolimbic areas.

Conscious hearing requires Conscious hearing requires activation of the auditory cortex . activation of the auditory cortex .

Books that teach improvisation Books that teach improvisation encourage imaging and hearing encourage imaging and hearing the effect of playing internally.the effect of playing internally.

Cognitive model of music processing: no imagery, no top-down processes.Cognitive model of music processing: no imagery, no top-down processes.Peretz I, Coltheart M, Modularity of music processing, Peretz I, Coltheart M, Modularity of music processing, Nature Neuroscience, vol. 6(7), 688-691, 2003; model also used in: Nature Neuroscience, vol. 6(7), 688-691, 2003; model also used in: Stewart L. et al. Music and the brain: disorders of musical listening. Stewart L. et al. Music and the brain: disorders of musical listening. Brain, 129, 2533-2553, 2006. Brain, 129, 2533-2553, 2006.

Music ImageryMusic ImageryMusic ImageryMusic ImageryfMRI hemodynamic increase during an Auditory Imagery Task performed in silence, in the auditory cortex posterior superior temporal gyrus.

Zatorre & Halpern, Mental Concerts: Musical Imagery and Auditory Cortex, Neuron 47, 9-12, 2004.

Aural imagery or inner hearing is considered to be an important aspect of musical development. Musicians need to connect the sound they desire with a "feel" they know will produce that sound. The goal of music performance is the reproduction of the internal auditory image. (D.R. Allen musicology thesis, 2007)

“An anticipatory image of feedback from an action participates in the selection and initiation or that action. [...] In the closed-loop formulation, the image may serve as a template for comparison with current feedback and need not be activated prior to performance.” A.G. Greenwald, Psych. Rev, 77, 73-99, 1970.

Music deficitsMusic deficitsMusic deficitsMusic deficitsStewart et. al, Music and the brain: disorders of musical listening. Brain, 129, 2533-2553, 2006, long review.

Pitch change directions, intervals, melodic patterns, contours, tonal structure, timbre, temporal structures (intervals, rhythm and meter), memory and emotional responses due to neurological problems are described.

Congenital amusia: true perceptual agnosia, although hearing and cognition is normal perception of music is not, usually deficit in pitch processing.

Mandell J, Schulze K, Schlaug G, Congenital amusia: An auditory-motor feedback disorder? Restorative Neurology and Neuroscience 25, 2007

“Thus, it is conceivable that individuals with congenital amusia, or the inability to sing in tune, may actually have an impairment of the auditory-motor feedback loop and/or auditory-motor mapping system.”

Conceivable, but some may have simply poor top-down feedback. This seems to be a condition that has not been clearly identified, a new kind of imagery amusia, the inability to imagine sounds.

Amusia and spatial processingAmusia and spatial processingAmusia and spatial processingAmusia and spatial processingAnatomical locus of amusia, neuroimaging/lesion studies: auditory areas along the superior temporal gyrus in pitch discrimination and melodic contour processing.

No evidence for morphological correlates of amusia in parietal regions. “The deficit may derive from changes in neural functioning that are invisible to the tools that have been applied to date.”

But amusics show deficits relative to musicians and nonmusicians, in tasks that involve spatial processing.

Douglas, K.M. & Bilkey, D.K. Nature Neurosci. 10, 915-921 (2007)

IQ and chronometryIQ and chronometryIQ and chronometryIQ and chronometryIn experiments that distinguish whether high/low tone was first 4 groups of people differening by their IQ showed clear correlation between % of errors depending on time intervals that changed from 5-300 ms.

J. Dreszer (UMK Torun),E. Szelag (Nencki Institute, Warsaw)

Parietal cortexParietal cortexParietal cortexParietal cortexA. Tosoni et al, Nature Neuroscience (Nov 2008 | doi:10.1038/nn.2221)Sensory-motor mechanisms in human parietal cortex underlie arbitrary visual decisions.

In arbitrary association of visual stimuli with different actions, activity of effector-specific regions in human posterior parietal cortex did not respond to sensory stimuli per se, but to integrated sensory evidence toward the decision outcome, triggered by contextual stimulus-response associations.

Hypothesis: normal perception requires top-down influences to form expectations. What if feedback connections to visual/auditory areas are weak?

C. Gilbert, M. Sigman, Brain States: Top-Down Influences in Sensory Processing.Neuron 54, 677-696, 2007.

“New findings on the diversity of top-down interactions show that cortical areas function as adaptive processors, being subject to attention, expectation, and perceptual task. Brain states are determined by the interactions between multiple cortical areas and the modulation of intrinsic circuits by feedback connections. ... Disruption of this interaction may lead to behavioral disorders.”

Imagery AgnosiaImagery AgnosiaImagery AgnosiaImagery AgnosiaNew branch of neuropsychology: imagery agnosias. Classical agnosias ~30 major types: alexia, akinetopsia, alexithimia, many visual types: prosopagnosia, simultanagnosia, semantic agnosia , form, color … Little access to perceptual imagery in visual, auditory, tactile or gustatory mode.Without internal feedback the only way to learn about plans formed by the brain is to act and observe results: trying to play an instrument in this condition is like blindsight, maneuvering blindly in the auditory space, without the ability to imagine results of next move (hitting piano key). •Learning to play music without imagery is difficult – how far can one go? Which key do I have to press if I have no idea how it will sound like? •Recognition memory is fine, but it is difficult to repeat or remember simple melodies (memory-motor map). •No problem to read & improvise music, higher cognition is fine. •Conscious mental rehearsal is not possible. •Immediate feedback may help?

The ListenerThe Listener

James C. Christensen

Extended mindExtended mindExtended mindExtended mindPhilosophers talk about parts of environment (notebooks for external memory etc) tightly coupled to our motor and cognitive activity as “extended mind”.

•Robert K. Logan, The Extended Mind: The Emergence of Language, the Human Mind and Culture. University of Toronto Press 2007

Inability to consciously interpret internal brain states leads to the need to express and recognize them through various bodily actions.

•David Kirsh, Interactive Cognition Lab at UCSD, thinking by doing.

•Understanding oneself = own brain states, allows for self-reflection and better self-regulation, but information flow in the brain is not perfect.

•Spontaneous thoughts are the main source of knowing, but acting out, motor activity also help to recognize brain processes (ex. humming melodies).

Imagery agnosia may be rather common (more common than classical agnosia).

•Investigation of this condition may elucidate relations between conscious and unconscious processes in implementation of higher cognitive functions.

Practical implicationsPractical implicationsPractical implicationsPractical implicationsBetter understanding of these issues will have far reaching implications for education, assessment of talent, understanding role of conscious experiences. •Work on VVIQ-like questionnaire-based evaluation of auditory imagery. •Repeat ERP experiments using NIR-OT (Janata, Zatorre) – imagery of missing sounds, priming cadences, look at auditory cortex response. •Development of simple tests for imagery agnosia, correlations of such tests with NIR-OT, ERP studies. Ex: Correlation of sounds with buttons: memory for mapping sounds to buttons should correlated with the ability to imagine sounds; as memory for images should correlate with ability to draw from memory. •Collecting statistical data, ex. children in music classes at school, correlation of IA with grades in different subjects? •Gender differences?

•Can one recover from imagery agnosia? Neurofeedback? Is it good goal?Influence of intensive musical training on auditory cortex activation. •Neuroimaging: top-down and bottom-up distinction?

Mental energyMental energyCan one train willpower, motivation, perseverance, curiosity and creativity using neurofeedback?Thinking and self-control requires mental energy derived from food. Higher levels of blood glucose reduce reliance on intuitive, heuristic-based decision making and help controlling attention, regulating emotions, coping with stress, resisting impulsivity, refraining from aggressive behavior. Alcohol reduces glucose levels impairing many forms of self-control. Self-control failure is most likely during times of the day when glucose is used least effectively.

Willpower and glucose•Gailliot MT & Baumeister RF (2007) The physiology of willpower: Linking blood glucose to self-control. Personality and Social Psychology Review, 11, 303-327. •Masicampo EJ & Baumeister RF (2008) Toward a physiology of dual-process reasoning and judgment: Lemonade, willpower, and effortful rule-based analysis. Psychological Science, 19, 255-260. Sufficient energy for mental activity is a necessary condition.

EEG and creativityEEG and creativityEEG and creativityEEG and creativityHow to increase cooperation between distant brain areas important for creativity?

John H. Gruzelier (Imperial College), SAN President neurofeedback produced “professionally significant performance improvements” in music and dance students. Neurofeedback and heart rate variability (HRV) biofeedback. benefited performance in different ways. Musicality of violin music students was enhanced; novice singers from London music colleges after ten sessions over two months learned significantly within and between session the EEG self-regulation of ratio. The pre-post assessment involved creativity measures in improvisation, a divergent production task, and the adaptation innovation inventory. Support for associations with creativity followed improvement in creativity assessment measures of singing performance. Why? 1.Low frequency waves = easier synchronization between distant areas. 2.Background processes and parasite oscillations decrease (can one see it with NIR-OT or EEG?).

Neurocognitive informaticsNeurocognitive informaticsNeurocognitive informaticsNeurocognitive informaticsUse inspirations from the brain, derive practical algorithms!

My own attempts - see my webpage, Google: W. Duch

1. Mind as a shadow of neurodynamics: geometrical model of mind processes, psychological spaces providing inner perspective as an approximation to neurodynamics.

2. Intuition: learning from partial observations, solving problems without explicit reasoning (and combinatorial complexity) in an intuitive way.

3. Neurocognitive linguistics: how to find neural pathways in the brain.

4. Creativity in limited domains + word games, good fields for testing.

Duch W, Intuition, Insight, Imagination and Creativity, IEEE Computational Intelligence Magazine 2(3), August 2007, pp. 40-52

IntuitionIntuitionIntuitionIntuitionIntuition is a concept difficult to grasp, but commonly believed to play important role in business and other decision making; „knowing without being able to explain how we know”.

Sinclair Ashkanasy (2005): intuition is a „non-sequential information-processing mode, with cognitive & affective elements, resulting in direct knowing without any use of conscious reasoning”.

3 tests measuring intuition: Rational-Experiential Inventory (REI), Myers-Briggs Type Inventory (MBTI) and Accumulated Clues Task (ACT).

Different intuition measures are not correlated, showing problems in constructing theoretical concept of intuition. Significant correlations were found between REI intuition scale and some measures of creativity.

ANNs evaluate intuitively? Yes, although intuition is used also in reasoning.

Intuition in chess has been studied in details (Newell, Simon 1975).

Intuition may result from implicit learning of complex similarity-based evaluation that are difficult to express in symbolic (logical) way.

Intuitive thinkingIntuitive thinking

Learning from partial observations: Ohm’s law V=I×R; Kirhoff’s V=V1+V2.

Geometric representation of qualitative facts:+ increasing, 0 constant, - decreasing.

True (I-,V-,R0), (I+,V+,R0), false (I+,V-,R0).

5 laws: 3 Ohm’s 2 Kirhoff’s laws.All laws A=B+C, A=B×C , A-1=B-1+C-1, have identical geometric interpretation!

13 true, 14 false facts; simple P-space, but complex neurodynamics.

Question in qualitative physics (PDP book): Question in qualitative physics (PDP book): if if RR22 increases, increases, RR11 and and VVtt are constant, what will are constant, what will happen with current and happen with current and VV11, , VV22 ??

Words in the brainWords in the brainWords in the brainWords in the brainPsycholinguistic experiments show that most likely categorical, phonological representations are used, not the acoustic input.Acoustic signal => phoneme => words => semantic concepts.Phonological processing precedes semantic by 90 ms (from N200 ERPs).F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press.

Left hemisphere: precise representations of symbols, including phonological components; right hemisphere? Sees clusters of concepts.

Action-perception networks inferred from ERP and fMRI

Neuroimaging wordsNeuroimaging words

Predicting Human Brain Activity Associated with the Meanings of Nouns," T. M. Mitchell et al, Science, 320, 1191, May 30, 2008

•Clear differences between fMRI brain activity when people read and think about different nouns.•Reading words and seeing the drawing invokes similar brain activations, presumably reflecting semantics of concepts.•Although individual variance is significant similar activations are found in brains of different people, a classifier may still be trained on pooled data. •Model trained on ~10 fMRI scans + very large corpus (1012) predicts brain activity for over 100 nouns for which fMRI has been done.

Overlaps between activation of the brain for different words may serve as expansion coefficients for word-activation basis set.

In future: I may know what you’ll think before you will know it yourself! Intentions may be known seconds before they become conscious!

ConnectomeConnectome Project ProjectConnectomeConnectome Project Project

Brain-based representations of concepts should be possible.

Nicole Speer et al. Reading Stories Activates Neural Representations of Visual and Motor Experiences. Psychological Science (2010, in print).Meaning: always slightly different, depending on the context, but still may be clusterized into relatively small number of distinct meanings.

Hidden conceptsHidden conceptsHidden conceptsHidden concepts• Language, symbols in the brain: phonological labels associated with protypes

of distributed activations of the brain.Helps to structure the flow of brain states in the thinking process. Do we have conscious access to all brain states that influence thinking? Right hemisphere activations just give us the feeling wrong something here.• Right hemisphere is as busy as left – concepts without verbal labels? • Evidence: insight phenomena, intuitive understanding of grammar, etc.

Can we describe verbally natural categories?• Yes, if they are rather distinct: see 20 question game.• Is object description in terms of properties sufficient and necessary? • Not always. Example: different animals and dog breeds.• 20Q-game: weak question (seemingly unrelated to the answer) may lead to

precise identification! RH may contribute to activation enabling associations

Dog breedsDog breedsDog breedsDog breeds

329 breeds in 10 categories:

Sheepdogs and Cattle Dogs; Pinscher and Schnauzer; Spitz and Primitive; Scenthounds; Pointing Dogs; Retrievers, Flushing Dogs and Water Dogs; Companion and Toy Dogs; Sighthounds

Write down properties and try to use them in the 20-question game to recognize the breed … fails!

Visually each category is quite different, all traditional categorizations are based on behavious and features that are not easy to observe.

• Ontologies do not agree with visual similarity.

• Brains discover it easily => not all brain states have linguistic labels.

Creativity & TriadsCreativity & TriadsCreativity & TriadsCreativity & Triads3 steps to solve problem:• Get information, prepare or prime your network (“search space”). • Spread this information around specialized circuits that enhance/modify it. • Filter the most relevant result.

Iteratively repeated search for solution constrained by knowledge that is already present in the network.

Blind-variation and selective-retention (Campbell 1960, Simonton 2010).

Creativity in word domain: assuming that the network has been trained on some language and has at least captured its statistical regularities:

• Prime the network with description of your product.• Imagery creates large number of novel constituents.• Filter using phonological and semantic density.

But what insight do I have into the result? Self filters the most relevant ones.Free will and action: self acts as a filter selecting alternative plans for actions.

Creativity with wordsCreativity with wordsCreativity with wordsCreativity with wordsThe simplest testable model of creativity: • create interesting novel words that capture some features of products;• understand new words that cannot be found in the dictionary.

Model inspired by the putative brain processes when new words are being invented starting from some keywords priming auditory cortex.

Phonemes (allophones) are resonances, ordered activation of phonemes will activate both known words as well as their combinations; context + inhibition in the winner-takes-most leaves only a few candidate words.

Creativity = network+imagination (fluctuations)+filtering (competition)

Imagination: chains of phonemes activate both word and non-word representations, depending on the strength of the synaptic connections. Filtering: based on associations, emotions, phonological/semantic density.

discoverity = {disc, disco, discover, verity} (discovery, creativity, verity)digventure ={dig, digital, venture, adventure} new! BrainGene server: http://www.braingene.yoyo.pl/ novel names, passwords …

Type Features

Dynamic Selection Network (DSN)

Critic

Idea

Descriptive Features

Concepts

System Connectivity

Context

Categories

Each activated concept corresponds to the resonant activity of a multi-level network.

Ali Minai architecture for creative associations, WCCI10

Problems requiring Problems requiring insightsinsights

Problems requiring Problems requiring insightsinsights

Given 31 dominos and a chessboard with 2 cornersremoved, can you cover all board with dominos?

Analytical solution: try all combinations.

Does not work … to many combinations to try.

Logical, symbolic approach has little chance to create proper activations in the brain, linking new ideas: otherwise there will be too many associations, making thinking difficult.

Insight <= right hemisphere, meta-level representations without phonological (symbolic) components ... counting?

d o

mi

no

phonological reps

chess board

blackwhite

domino

Insights and brainsInsights and brainsInsights and brainsInsights and brainsActivity of the brain while solving problems that required insight and that could be solved in schematic, sequential way has been investigated. E.M. Bowden, M. Jung-Beeman, J. Fleck, J. Kounios, „New approaches to demystifying insight”. Trends in Cognitive Science 2005.

After solving a problem presented in a verbal way subjects indicated themselves whether they had an insight or not.

An increased activity of the right hemisphere anterior superior temporal gyrus (RH-aSTG) was observed during initial solving efforts and insights. About 300 ms before insight a burst of gamma activity was observed, interpreted by the authors as „making connections across distantly related information during comprehension ... that allow them to see connections that previously eluded them”.

Insight interpretedInsight interpretedInsight interpretedInsight interpreted

What really happens? My interpretation:

• LH-STG represents concepts, S=Start, F=final• understanding, solving = transition, step by step, from S to F• if no connection (transition) is found this leads to an impasse; • RH-STG ‘sees’ LH activity on meta-level, clustering concepts into abstract

categories (cosets, or constrained sets);• connection between S to F is found in RH, leading to a feeling of vague

understanding; • gamma burst increases the activity of LH representations for S, F and

intermediate configurations; feeling of imminent solution arises;• stepwise transition between S and F is found;• finding solution is rewarded by emotions during Aha! experience;

they are necessary to increase plasticity and create permanent links.

Solving problems with Solving problems with insightinsight

Solving problems with Solving problems with insightinsight

Right temporallobe

Left temporal lobe

Start: problem statement

Goal

Steps

Neuromodulation (emotions)

Mental modelsMental modelsMental modelsMental models

P. Johnson-Laird, 1983 book and papers. Imagination: mental rotation, time ~ angle, about 60o/sec.Internal models of relations between objects, hypothesized to play a major role in cognition and decision-making. AI: direct representations are very useful, direct in some aspects only!

Reasoning: imaging relations, “seeing” mental picture, semantic? Systematic fallacies: a sort of cognitive illusions.

•If the test is to continue then the turbine must be rotating fast enough to generate emergency electricity.•The turbine is not rotating fast enough to generate this electricity.•What, if anything, follows? Chernobyl disaster …

If A=>B; then ~B => ~A, but only about 2/3 students answer correctly.

Kenneth Craik, 1943 book “The Nature of Explanation”, G-H Luquet attributed mental models to children in 1927.

Mental models summaryMental models summaryMental models summaryMental models summary

1. MM represent explicitly what is true, but not what is false; this may lead naive reasoner into systematic error.

2. Large number of complex models => poor performance. 3. Tendency to focus on a few possible models => erroneous conclusions and irrational

decisions.

Cognitive illusions are just like visual illusions.M. Piattelli-Palmarini, Inevitable Illusions: How Mistakes of Reason Rule Our Minds (1996)R. Pohl, Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement and

Memory (2005)

Amazing, but mental models theory ignores everything we know aboutlearning in any form! How and why do we reason the way we do? I’m innocent! My brain made me do it!

The mental model theory is an alternative to the view that deduction depends on formal rules of inference.

Trajectories for psycholinguisticsTrajectories for psycholinguisticsP.McLeod, T. Shallice, D.C. Plaut, Attractor dynamics in word recognition: converging evidence from errors by normal subjects, dyslexic patients and a connectionist model. Cognition 74 (2000) 91-113.

New area in psycholinguistics: investigation of dynamical cognition, influence of masking on semantic and phonological errors.

Model Model of reading & dyslexiaof reading & dyslexia

Learning: mapping one of the 3 layers to the other two.Fluctuations around final configuration = attractors representing concepts.How to see properties of their basins, their relations?

Emergent neural simulator:Aisa, B., Mingus, B., and O'Reilly, R. The emergent neural modeling system. Neural Networks,

21, 1045-1212, 2008.

3-layer model of reading: orthography, phonology, semantics, or distribution of activity over 140 microfeatures of concepts. Hidden layers in between.

P-spacesP-spacesP-spacesP-spacesPsychological spaces: how to visualize inner life?

K. Lewin, The conceptual representation and the measurement of psychological forces (1938), cognitive dynamic movement in phenomenological space.

George Kelly (1955): personal construct psychology (PCP), geometry of psychological spaces as alternative to logic.

A complete theory of cognition, action, learning and intention.

PCP network, society, journal, software … quite active group.

Many things in philosophy, dynamics, neuroscience and psychology, searching for new ways of understanding cognition, are relevant here.

P-space definitionP-space definitionP-space definitionP-space definition

P-space: region in which we may place and classify elements of our experience, constructed and evolving, „a space without distance”, divided by dichotomies.

P-spaces should have (Shepard 1957-2001):

• minimal dimensionality;• distances that monotonically decrease with increasing similarity.

This may be achieved using multi-dimensional non-metric scaling (MDS), reproducing similarity relations in low-dimensional spaces.

Many Object Recognition and Perceptual Categorization models assume that objects are represented in a multidimensional psychological space; similarity between objects ~ 1/distance in this space.

Can one describe the state of mind in similar way?

AttractorsAttractorsAttention results from: • inhibitory competition, • bidirectional interactive processing, • multiple constraint satisfaction.

Basins of attractors: input activations {LGN(X)}=> object recognition

• Normal case: relatively large, easy associations, moving from one basin of attraction to another, exploring the activation space.

• Without accommodation (voltage-dependent K+ channels): deep, narrow basins, hard to move out of the basin, associations are weak.

Accommodation: basins of attractors shrink and vanish because neurons desynchronize due to the fatigue; this allows other neurons to synchronize, leading to quite unrelated concepts (thoughts).

Recurrence plotsRecurrence plots

Same trajectories displayed with recurrence plots, showing roughly 5 larger basins of attractors and some transient points.

Starting from the word “flag”, with small synaptic noise (var=0.02), the network starts from reaching an attractor and moves to another one (frequently quite distant), creating a “chain of thoughts”.

Normal-ADHDNormal-ADHD

All plots for the flag word, different values of b_inc_dt parameter in the accommodation mechanism. b_inc_dt = 0.01 & b_inc_dt = 0.02b_inc_dt = time constant for increases in intracellular calcium which builds up slowly as a function of activation. http://kdobosz.wikidot.com/dyslexia-accommodation-parameters

Normal-AutismNormal-Autism

All plots for the flag word, different values of b_inc_dt parameter in the accommodation mechanism. b_inc_dt = 0.01 & b_inc_dt = 0.005b_inc_dt = time constant for increases in intracellular calcium which builds up slowly as a function of activation. http://kdobosz.wikidot.com/dyslexia-accommodation-parameters

Neurocognitive reps.Neurocognitive reps.How to approach modeling of word (concept) w representations in the brain? Word w = (wf,ws) has

•phonological (+visual) component wf, word form;

•extended semantic representation ws, word meaning;

•is always defined in some context Cont (enactive approach).

(w,Cont,t) evolving prob. distribution (pdf) of brain activations.Hearing or thinking a word w , or seeing an object labeled as w adds to the overall brain activation in a non-linear way.

How? Spreading activation in neural spaces, maximizing overall self-consistency, mutual activations, meanings that don’t fit to current context are automatically inhibited. Result: almost continuous variation of this meaning.

This process is rather difficult to approximate using typical knowledge representation techniques, such as connectionist models, semantic networks, frames or probabilistic networks.

ApproximateApproximate reps. reps.States (w,Cont) lexicographical meanings:

•clusterize (w,Cont) for all contexts;

•define prototypes (wk,Cont) for different meanings wk.

A1: use spreading activation in semantic networks to define . A2: take a snapshot of activation in discrete space (vector approach).

Meaning of the word is a result of priming, spreading activation to speech, motor and associative brain areas, creating affordances.

(w,Cont) ~ quasi-stationary wave, with phonological/visual core activations wf and variable extended representation ws selected by Cont.

(w,Cont) state into components, because the semantic representation

E. Schrödinger (1935): best possible knowledge of a whole does not include the best possible knowledge of its parts! Not only in quantum case. Left semantic network LH contains wf coupled with the RH.

QM-like formalism is useful for any probability waves.

Semantic => vector repsSemantic => vector repsSome associations are subjective, some are universal. How to find the activation pathways in the brain? Try this algorithm:

• Perform text pre-processing steps: stemming, stop-list, spell-checking ...• Map text to some ontology to discover concepts (ex. UMLS ontology). • Use relations (Wordnet, ULMS), selecting those types only that help to

distinguish between concepts.• Create first-order cosets (terms + all new terms from included relations),

expanding the space – acts like a set of filters that evaluate various aspects of concepts.

• Use feature ranking to reduce dimensionality of the first-order coset space, leave all original features.

• Repeat last two steps iteratively to create second- and higher-order enhanced spaces, first expanding, then shrinking the space.

Result: a set of X vectors representing concepts in enhanced spaces, partially including effects of spreading activation.

How to become an expertHow to become an expert??Textbook knowledge in medicine: detailed description of all possibilities.Effect: neural activation flows everywhere and correct diagnosis is impossible. Correlations between observations forming prototypes are not firmly established. Expert has correct associations.Example: 3 diseases, clinical case description, MDS description.1)System that has been trained on textbook knowledge.2)Same system that has learned on real cases.3)Experienced expert that has learned on real cases.

Conclusion: abstract presentation of knowledge in complex domains leads to poor expertise, random real case learning is a bit better, learning with real cases that cover the whole spectrum of different cases is the best.

I hear and I forget.I see and I remember.I do and I understand.

Confucius, -500 r.

Humanized interface

Store

Applications, search, 20 questions game.

Query

Semantic memory

Parser

Part of speech tagger& phrase extractor

On line dictionariesActive search and dialogues with usersManual

verification

DREAM top-level architectureDREAM top-level architecture

Natural input modules

Cognitive functions

Affectivefunctions

Web/text/databases interface

Behavior control

Control of devices

Talking head

Text to speechNLP

functions

Specializedagents

DREAM project is focused on perception (visual, auditory, text inputs), cognitive functions (reasoning based on perceptions), natural language communication in well defined contexts, real time control of the simulated/physical head.

DREAM/HIT – larger view … DREAM/HIT – larger view … DREAM/HIT – larger view … DREAM/HIT – larger view …

HIT projectsHIT projects

T-T-S synthesisT-T-S synthesis

Speech recognitionSpeech recognition

Talking headsTalking heads

BehavioralBehavioralmodelsmodels

GraphicsGraphics

Cognitive ArchitecturesCognitive Architectures

Cognitive Cognitive sciencescience

AIAI

A-MindsA-MindsLingu-botsLingu-bots

KnowledgeKnowledgemodelingmodelingInfo-retrievalInfo-retrieval

VR avatarsVR avatars

RoboticsRobotics

Brain modelsBrain models

Affective Affective computingcomputing

EpisodicEpisodicMemoryMemorySemantic Semantic

memorymemory

WorkingWorkingMemoryMemory

LearningLearning

ConnectionsConnectionsConnectionsConnections

Geometric/dynamical ideas related to mind may be found in many fields:

Neuroscience: D. Marr (1970) “probabilistic landscape”. C.H. Anderson, D.C. van Essen (1994): Superior Colliculus PDF mapsS. Edelman: “neural spaces”, object recognition, global representation space approximates the Cartesian product of spaces that code object fragments, representation of similarities is sufficient.

Psychology: K. Levin, psychological forces.G. Kelly, Personal Construct Psychology.R. Shepard, universal invariant laws.P. Johnson-Laird, mind models.

Folk psychology: to put in mind, to have in mind, to keep in mind (mindmap), to make up one's mind, be of one mind ... (space).

More connections More connections More connections More connections AI: problem spaces - reasoning, problem solving, SOAR, ACT-R, little work on continuous mappings (MacLennan) instead of symbols.

Engineering: system identification, internal models inferred from input/output observations – this may be done without any parametric assumptions if a number of identical neural modules are used!

Philosophy: P. Gärdenfors, Conceptual spacesR.F. Port, T. van Gelder, ed. Mind as motion (MIT Press 1995)

Linguistics: G. Fauconnier, Mental Spaces (Cambridge U.P. 1994).

Mental spaces and non-classical feature spaces. J. Elman, Language as a dynamical system; J. Feldman neural basis;

Stream of thoughts, sentence as a trajectory in P-space.

Psycholinguistics: T. Landauer, S. Dumais, Latent Semantic Analysis, Psych. Rev. (1997) Semantic for 60 k words corpus requires about 300 dim.

Some questionsSome questionsSome questionsSome questionsTo understand talent in its many forms it would be good to know:

• Many types of agnosia exist: do all have their imagery versions? • Statistics on strong visualizers/ auditory/taste imagery ect. How strong?

Is there correlation between lack of different types of imagery? • 15% of population has difficulty in singing, but only 5% congenitial amusia.• Interpretation of many experiments may be wrong, mixing brains that work

in quite different styles (compare statistics for cognitive decline).

• High-level processes, such as imagery, planning & creativity, do not have to be conscious – when is consciousness essential and how does it help?

• Many important brain science questions: averaging in fMRI over different brain processing styles; spatial deficits correlation with amusia => try to distinguish between imagery and other types of amusia and find people with amusia without spatial deficits!

Possible applicationsPossible applicationsPossible applicationsPossible applicationsTwo main areas: Education and Brain-Computer Interfaces (BCI).

Education•A simple giftedness tests based on ERPs or NIRS-OT for imagery? •Music/art education only? Abstract thinking? Thinking by doing (Kirsch)? •How to develop personalized education styles? Enhance curiosity, imagination, motivation, perseverance, strong will, creativity …

Brain-Computer Interfaces & Neurofeedback•BCI based on sound imagery (with Klonowski, Perovic, Jovanovic). •How many states can be distinguished by pitch? Timbre? Rhythm imagery? •Neurofeedback based on synchronization/connectivity assessment.Klonowski W, Duch W, Perovic A, Jovanovic A, Some Computational Aspects of the Brain Computer Interfaces Based on Inner Music. Computational Intelligence and Neuroscience, Vol 2009, Article ID 950403, 9 pages, 2009. doi:10.1155/2009/950403

• Phantomology: the virtual reality of the body. Theoretical aspects and hands-on demonstrations (co-organized with Dept. of Neurology, University Hospital Zurich), with Olaf Blanke, Peter Brugger, Henrik Ehrsson, Kevin O’Regan and others (second half of Sept 2011).

• Brain and music, May 2011 (probably 22-24.05), with J. LeDoux ... • Crossmodality and synesthesia (2012)

Exciting times Exciting times are coming! are coming!

Thank you Thank you for for

synchronizing synchronizing your neurons your neurons and lending and lending your ears your ears

Google: W Duch => PapersGoogle: W Duch => Papers