Neural network applications: Neural network applications: The present and the futureThe present and the future

Neural network applications: Neural network applications: The present and the futureThe present and the future

Włodzisław Duch

Department of Informatics, Nicolaus Copernicus University, Toruń, Poland

Google: W. DuchICONIP’08 Panel Discussion

In the year 1900 at the International Congress of Mathematicians in Paris David Hilbert delivered what is now considered the most important talk ever given in the history of mathematics, proposing 23 major problems worth working at in future. 100 years later the impact of this talk is still strong: some problems have been solved, new problems have been added, but the direction once set - identify the most important problems and focus on them - is still important.

It became quite obvious that this new field also requires a series of challenging problems that will give it a sense of direction.

• Wlodzislaw Duch, What Is Computational Intelligence and Where Is It Going?

• Jurgen Schmidhuber, New Millennium AI and the Convergence of History

• Ron Sun, The Challenges of Building Computational Cognitive Architectures

• James A. Anderson et al. Programming a Parallel Computer: The Ersatz Brain Project

• JG Taylor, The Human Brain as a Hierarchical Intelligent Control System

• Soo-Young Lee, Artificial Brain and OfficeMateTR based on Brain Information Processing Mechanism

• Stan Gielen, Natural Intelligence and Artificial Intelligence: Bridging the Gap between Neurons and Neuro-Imaging to Understand Intelligent Behaviour

• DeLiang Wang, Computational Scene Analysis

• Nikola Kasabov, Brain-, Gene-, and Quantum Inspired Computational Intelligence: Challenges and Opportunities

• Robert P.W. Duin, Elżbieta Pękalska, The Science of Pattern Recognition. Achievements and Perspectives

• Wlodzislaw Duch, Towards Comprehensive Foundations of Computational Intelligence

• Witold Pedrycz, Knowledge-Based Clustering in Computational Intelligence

• Vera Kurkova, Generalization in Learning from Examples

• Lei Xu, A Trend on Regularization and Model Selection in Statistical Learning: A Bayesian Ying Yang Learning Perspective

• Jacek Mańdziuk, Computational Intelligence in Mind Games

• Xindi Cai and Donald C. Wunsch II, Computer Go: A Grand Challenge to AI

• Lipo Wang and Haixiang Shi, Noisy Chaotic Neural Networks for Combinatorial Optimization

Grand challengesGrand challengesGrand challengesGrand challengesOur discipline is broad, and there many grand challenges for the next 20 years.

• Foundations for CI theory, integrating all methods.• Learning from data in difficult cases• Complex models, structured data, natural perception• Understanding brain/mind relations, neuromorphic models• Natural language processing• Combining CI (perception) with AI (systematic reasoning)• Towards artificial minds

Artificial Minds (AMs), or personoids, are software and robotic agents

that humans can talk to and relate to in a similar way as they relate to

other humans.

Neurocognitive informatics!

Current projectsCurrent projects Current projectsCurrent projects

• Learning data with inherent complex logic, general theory of CI and meta-learning.

• Infant lab for developing perfect babies, testing for problems/talents, and other neuroengineering projects – observing real behavior and understanding these observations.

• Understanding real brains: breaking neural code, brain stem model, priming in cortex, generative disease models.

• Brain-inspired cognitive architectures, avatars with artificial minds, emotions, creativity & hi-level cognition.

• Neurocognitive inspirations in natural language processing, large scale semantic memories, word games, structuring information, precisiation of queries, semantic web, text annotation, bibliography, literature-based discovery.

• Interactive art projects, computer games.

What is there to learn?What is there to learn?What is there to learn?What is there to learn?Brains ... what is in EEG? What happens in the brain?

Industry: what happens?

Genetics, proteins ...

What can we learn?What can we learn?What can we learn?What can we learn?Good part of CI is about learning.What can we learn?

Neural networks are universal approximators and evolutionary algorithms solve global optimization problems – so everything can be learned? Not quite ...

Duda, Hart & Stork, Ch. 9, No Free Lunch + Ugly Duckling Theorems:

• Uniformly averaged over all target functions the expected error for all learning algorithms [predictions by economists] is the same. • Averaged over all target functions no learning algorithm yields generalization error that is superior to any other. • There is no problem-independent or “best” set of features.

“Experience with a broad range of techniques is the best insurance for solving arbitrary new classification problems.”

Data mining packagesData mining packagesData mining packagesData mining packages

• DM packages: Weka, Yale, RapidMiner, Orange, Knime ... >180 packages on the-data-mine.com list!

Hundreds of components ... thousands of combinations ... Our treasure box is full, although computer vision, BCI

and other problems are not solved.

• We can data mine forever … and publish forever!

• Neural networks are universal approximators and evolutionary algorithms solve global optimization problems – so everything can be learned? Not quite ...

GhostMiner, data mining tools from our lab + Fujitsu: http://www.fqspl.com.pl/ghostminer/

Are we really Are we really so good?so good?

Surprise!

Almost nothing can be learned using such tools!

What have we tried: SBMWhat have we tried: SBMWhat have we tried: SBMWhat have we tried: SBMSimilarity-Based Methods (SBMs) organized in a framework: p(Ci|X;M) posterior classification probability or y(X;M) approximators,models M are parameterized in increasingly sophisticated way.

Why? (Dis)similarity: • more general than feature-based description, • no need for vector spaces (structured objects), • more general than fuzzy approach (F-rules are reduced to P-rules), • includes kNN, MLPs, RBFs, separable function networks, SVMs,

kernel methods and many others!

Components => Models; systematic search selects optimal combination of parameters and procedures, opening different types of optimization channels, trying to discover appropriate bias for a given problem.

Start from kNN, k=1, all data & features, Euclidean distance, end with a model that is a novel combination of procedures and parameterizations.

Transformation-based frameworkTransformation-based frameworkTransformation-based frameworkTransformation-based framework

Extend SBM adding fine granulation of methods and relations between them to enable meta-learning by search in the model space.

For example, transformations (layers) frequently do:

•linear projection: unsupervised - PCA, ICA … or supervised – FDA, LDA, linear SVM generate useful linear components;

•non-linear preprocessing transformation, ex. MLP;

•feature selector, based on information filter;

•matching pursuit network for signal decomposition;

•logical rules to handle unusual situations;

•evaluate similarity (RBF).

DM requires more transformations!

More meta-learningMore meta-learningMore meta-learningMore meta-learning

Meta-learning: learning how to learn, replace experts who search for best models making a lot of experiments.Search space of models is too large to explore it exhaustively, design system architecture to support knowledge-based search.

• Abstract view, uniform I/O, uniform results management.

• Directed acyclic graphs (DAG) of boxes representing scheme

• placeholders and particular models, interconnected through I/O.

• Configuration level for meta-schemes, expanded at runtime level.

An exercise in software engineering for data mining!

Intemi, Intelligent MinerIntemi, Intelligent MinerIntemi, Intelligent MinerIntemi, Intelligent MinerMeta-schemes: templates with placeholders

• May be nested; the role decided by the input/output types.

• Machine learning generators based on meta-schemes.

• Granulation level allows to create novel methods.

• Complexity control: Length + log(time)

• A unified meta-parameters description ...

• InteMi, intelligent miner, coming “soon”.

How much can we learn?How much can we learn?Linearly separable or almost separable problems are relatively simple – deform or add dimensions to make data separable.

How to define “slightly non-separable”? There is only separable and the vast realm of the rest.

Spying on networksSpying on networksSpying on networksSpying on networksAfter initial transformation, what still needs to be done?

Conclusion: separability in the hidden space is perhaps too much to desire ... rules, similarity or linear separation, depending on the case.

Parity n=9Parity n=9Parity n=9Parity n=9

Simple gradient learning; quality index shown below.