Total cognitive care forTotal cognitive care fordevelopment of perfect baby brainsdevelopment of perfect baby brains

Włodzisław DuchWłodzisław Duch

Dept. of Informatics, Nicolaus Copernicus University, PolandDept. of Informatics, Nicolaus Copernicus University, PolandDept. of Computer Science, School of Computer Engineering, Dept. of Computer Science, School of Computer Engineering,

Nanyang Technological University, SingaporeNanyang Technological University, Singapore

Google: DuchGoogle: Duch

PlanPlan

• Background: cognitive infant developmentBackground: cognitive infant development• Few science factsFew science facts• Total cognitive infant careTotal cognitive infant care• Toys that make you smartToys that make you smart• Techno remarksTechno remarks• Plans & strategy of developmentPlans & strategy of development• Endless possibilities ... Endless possibilities ...

Developmental problemsDevelopmental problems• Brain is the most complex organ and frequently not functioning well.

• About 5-10% of all children have a developmental disability that causes a delay in their speech and language development.

• If serious hearing problems are not found in the first six months of life, and the baby given the appropriate treatment, that child will grow up with significant damage to his or her language abilities.

• Children with normal cognitive development whose hearing losses are identified before six months can develop language at the same or a similar rate to a hearing child. Identification of congenital hearing loss in USA is at 2½ years of age!

• The British Dyslexia Association estimates that 10% of children have some degree of dyslexia, while about 4% will be affected severely (an average of one in every class).

Cognitive developmentCognitive development• Brain: the ultimate engineering problem! How to improve it?

• Although genes set a limit for potential individual development this limit is never reached; genetic manipultion is dangerous. Early development is critical for unfolding brain’s potential.

• Challenge: prevent abnormalities and boost normal development.

• Since genes specify only roughly where neurons should go (ex: eye to visual cortex, across the whole brain) too many neurons are created, and those unused die (apoptosis). Maximum number of neurons: 1-2 month before birth, although infant’s brain is only ¼ of the final size.

• General principle: growing up is specializing = narrowing potential possibilities. How to keep more possibilities open?

Example: motor developmentExample: motor development• Duke University, NC, 2002: velcro-covered mittens help

2-3 month infants develop more quickly.

• Infants are no yet able to grasp objects, but velcro mittens help them to catch and then touch toys;this speeds up motor coordination and perceptual development.

• Babies who had experience with the mittens outperformed the babies who didn't in a number of ways.

• Long-term consequences of this procedure are investigated by psychologists. News shown by CNN and elsewhere: in this area it is easy to get a free press ...

• Lynn-Flynn effect: IQ grows everywhere in the world, 24 points in USA since 1918, 27 points in UK.Toys and nutrition help to develop better brains?

Infant communicationInfant communication• Brain is capable of learning, but poor sensory analysis and motor

coordination makes it difficult to communicate with infants.• High-amplitude suction-methods are used with newborns and

infants to find how much interest they have in different stimuli.

• Sign-language for communication with infants (US patent): using simple signs with babies to bridge the gap between understanding and speech development

• Recommendations of developmental psychologists: pre-natal stimulation: BabyPlus™ “enhanced heartbeat”, playing music to babies in the womb. Playing tapes with foreign language lullabies etc.

• All this is passive learning, much less effective than interactive learning in which the infant is actively engaged. Can one learn walking just by observation?

• Learning to perceive, speaking and drawing inferences should be active, guided by well-structured inputs and positive feedback.

Speech perception factsSpeech perception facts• Speech perception is based on syllables, combinations of

elementary phonemes; 2-month old recognize syllables. • There are about 6800 languages, but only 800 phonemes.

• The ability to hear phonetic contrasts of all possible human languages is in-born, 6 month old are good in any language.

• This ability declines sharply 10 month after birth, only the ability to distinguish contrast in languages that are spoken to the child is preserved – one of the earliest specializations.

• At 5 month children are able to categorize a vowel in a speaker-independent way.

• Learning many languages correlates positively with IQ• Tonal languages, prevalent in Asia, are phonetically rich,

but even ‘simple’ /ra/-/la/ contrast is impossible for Japanese, /vi/-/bi/ for Spanish, /s/-/th/ for most non-English native speakers.

PhonemesPhonemesVowels spoken by 97 speakers, seen in the first 2 formants only.

Mapping to the auditory cortex looks similar.

Categorical perception means that what we hear jumps from a to e with no gradation between. ma – ma diff?

Speech problemsSpeech problems• Up to 4 million primary and secondary school students in USA

have difficulty distinguishing between phonemes, particularly between consonants like b, d and p; is it pee or bee, pun or bun?

• “Fast ForWord”® software by Scientific Learning helps to develop learning skills for 4-7 year olds. It is used in the Chicago public school system.

• Private clinicians provide Fast ForWord training (>$2,500).It shows remarkable success with kids who suffer from central auditory processing disorder that leads to speech understanding problems and retards reading.

• Big market: Learning Company, the producer of Reader Rabbit, was acquired by Mattel for $3.5 billion.

• Can we eliminate hearing and speech problems improving perceptual and cognitive skills of babies at the same time?

Active stimulator of brain’s speech centers, Active stimulator of brain’s speech centers, especially infant and childrenespecially infant and children

Application no: 184102, date: 29.07.1997, granted 30.08.2002Application no: 184102, date: 29.07.1997, granted 30.08.2002Creator and the sole rights holder: Duch Wlodzislaw, Torun, PLCreator and the sole rights holder: Duch Wlodzislaw, Torun, PL

Active stimulator of brain’s speech centers, especially infant and Active stimulator of brain’s speech centers, especially infant and children, with input device, analog-digital converter, memory children, with input device, analog-digital converter, memory containing phonemic patterns, comparator of phonemes received containing phonemic patterns, comparator of phonemes received through inputs and retrieved from the memory, logical decision unit, through inputs and retrieved from the memory, logical decision unit, digital-analog converter, and the output device for sound. The input digital-analog converter, and the output device for sound. The input device connects to the analog-digital converter, this converter is device connects to the analog-digital converter, this converter is connected to one input of the comparator, another input of the connected to one input of the comparator, another input of the comparator connects to the memory containing phonemic patterns, comparator connects to the memory containing phonemic patterns, and the comparator’s output is connected to a logical decision unit; and the comparator’s output is connected to a logical decision unit; the output from this unit connects with the sound producing device the output from this unit connects with the sound producing device through the digital-analog converter. through the digital-analog converter.

US5893720 abstractUS5893720 abstractHannah R Cohen (US) 1999 patent:Hannah R Cohen (US) 1999 patent:

A computer toy for infants that promotes normal speech development by facilitating the infant's experimentation with babbles and other elementary sounds. Additionally, the toy provides an enriched environment for language learning by prompting the infant with a repertoire of verbal sounds including phonemes, syllables, and simple spoken words.

The toy includes a microphone input device for detecting vocalizations by an infant and an audiovisual output device for providing feedback to the infant including the immediate playback of the infant's own vocalizations, and a control means for transforming detected vocalizations into instructions for use by the audiovisual output device.

Philips “Magic Mirror” is based on this principle.

Philips “Magic Mirror”• Some companies are going in this direction, making interactive Some companies are going in this direction, making interactive

toys, for example: toys, for example:

“Magic Mirror” introduced in 2003

Records and replays baby’s voice, triggers first speech sounds and has mirror, pushing buttons produces some sounds, encourages experimentation.

It is not aimed at enhancing specific perceptual skills.

Total cognitive careTotal cognitive care• Observe the baby and interpret his/her behavior by monitoring

vocalization, sucking response, movements, GSR etc.• Challenge the baby to solve perceptual/abstract problems.

Use natural audio-visual-tactile stimulation.• Reward behaviors that correlate with differences in stimuli.

For example, if a series of sounds is played:“la la la la ra ra ra ra”, reaction to la-ra change shows that the baby has noticed an important phonetic contrast.

• PerCog devices for enhancement and therapy of perceptual and cognitive skills use intelligent AI control to model what the baby has already learned, what and how often should be presented, observing, diagnosing, correcting and teaching infants.

• Hypothesis: active learning should gently pressure baby’s brain to develop perceptual/cognitive skills in the desired direction.

Toys for speech developmentToys for speech development• Goal: enhance phonematic hearing.

• Prepare a database of phonemes and syllables, present pairs

that are similar, and if the infant notices phonetic contrasts, apply positive stimulation. Start with basic contrasts, end with subtle.

• The infant is not able to repeat speech sounds: changes in the larynx, highly sophisticated control over vocal cords, are slower than neural development.

• Hypothesis: 1) Early (~10 month) specialization in correct discrimination of mother tongue phonetic contrast may be eliminated, giving the baby ability to learn any language she/he chooses.2) Sharper discrimination between basic phonemes should prevent some hearing and speech problems.

Telemetric pacifier

Control unit la-la … la-ra-ra… sound sequences

Database of speech sounds A/D converter receiver

Speaker

Audiovisual device (reward)

D/A converter

Wireless communication

Non-volatile memory

Database of reward patterns

RAM

Toys for musical earToys for musical ear• Goal: enhance musical hearing, develop perfect pitch.

• Only one person in 10.000 has perfect pitch; adult professional musicians try to learn it but it is very hard.

• Prepare a database of musical sounds of different instruments at different pitch, present pairs of sound samples with few seconds of silence in between, reward for signaling that sounds were of the same pitch although tones may differ. Start with pure frequencies, move to more complex sounds and accords of several sounds.

• Hypothesis: memory for absolute sound pitch should develop, increasing the ability to hear subtle musical structures.

Toys for abstract thinkingToys for abstract thinking• Goal: enhance the ability to think!

• Infants habituated for 2 minutes with sentences like ga ti ga, li na li, of the ABA structure, recognize that wo fe wo has correct grammatical structure but wo wo fe does not.

G. Marcus G. Marcus et alet al, “Rule learning by seven-month-old infants”, , “Rule learning by seven-month-old infants”, Science 1999, Vol. 283, pp. 77 – 80.Science 1999, Vol. 283, pp. 77 – 80.

• Challenge the infant using sounds and color lights with structures of increasing complexity, reward for noticing differences.

• Hypothesis: solving problems of this kind has strong influence on development of the cortex, in particular frontal and temporal areas, and should increase working memory span, that correlates well with general IQ.

More PerCog devicesMore PerCog devices

Goal: enhance perceptual discrimination in all modalities, improve categorical perception, increase working memory span, long-term perceptual memory, encourage abstractions and faster reactions.Responses to stimuli, behavioral interaction patterns should allow for early diagnosis of developmental problems.

Imagine ... Imagine ... • A world in which people could communicate A world in which people could communicate

without problems and even adults learn languages easily.without problems and even adults learn languages easily.

• A world in which developmental abnormalities related to speech A world in which developmental abnormalities related to speech and reading would largely vanish.and reading would largely vanish.

• A world much more rich in subtle sensory experiences.A world much more rich in subtle sensory experiences.

• A world in which the potential of each child would always fully A world in which the potential of each child would always fully develop, and natural curiosity will grow ... develop, and natural curiosity will grow ...

• We just have to start quite early, adding structure to the We just have to start quite early, adding structure to the development of the brain – in a natural, effortless way. development of the brain – in a natural, effortless way.

• This is possible with PerCog devices, such as cognitive toys!This is possible with PerCog devices, such as cognitive toys!• Guess what is the market for such devices? Guess what is the market for such devices?

So farSo far• Polish patent application was restricted to phoneme perception, the

present proposal is much wider and differs in important aspects.

• We have made some research assuming that the main feedback from the infant will be via vocalization: phonemes from 28 languages have been collected, software to present them, recognize change from silence/speech and vowel/consonant change in baby’s vocalization has been used; this is done by microphone and speaker connected to PC sound card.

• Sucking (HAS), head turning preference, visual fixation and other procedures used by psychologists for infants < 6 month are better.

• Some work on hardware implementation has been done, but it still requires a PC to analyze the data; hardware working prototype due to technical problems.

RoadmapRoadmap• First personalized medicine device, not related to molecular level:

adapts to the individual needs/abilities.

• Patent application: “Devices for enhancement and therapy of perceptual and cognitive skills of infants and children” for all kinds of PerCog devices and total cognitive care of infants.

• Ready for quick commercialization and more R&D. Largest market for speech devices is in the Far East, in the USA more prevention/diagnosis/treatment of speech/reading problems.

• Construction of PerCog devices in form of electronic toys or crib environments, initially with a modest number of phonemes covering basic phonetic contrasts, is not difficult.

• Computer games for babies and children may be created using the same principle.

Research opportunitiesResearch opportunities• Interdisciplinary group including experts in developmental

psychology, phonetics, neurolinguistics, speech/hearing/language neurolinguistics, speech/hearing/language pathologist, education experts, signal processing, pathologist, education experts, signal processing, electronics (sensors + IC), toy design, software development, data mining ... , data mining ...

• Creation of larger phonetic databases: most important phonetic contrasts for major languages, syllables and words which are hard to distinguish, like van/ban, thick/sick, extending it to cover subtle differences, and finally full spectrum of speech sounds.

• Improvement of the feedback recognition systems: wireless sucking monitoring devices, analysis of infant babbling for sounds that infants may use for signaling, observation of movements.

• Follow-up studies to investigate the effect of using cognitive speech Follow-up studies to investigate the effect of using cognitive speech toys on phonetic contrast perception (a few month), language toys on phonetic contrast perception (a few month), language learning (long-term studies) and reduction of speech and reading learning (long-term studies) and reduction of speech and reading abnormalities (long-term). abnormalities (long-term).

Brave new world?Brave new world?New generation of children may grow up to be: New generation of children may grow up to be:

much more intelligent than we aremuch more intelligent than we arethink much faster than we dothink much faster than we dobe much more musicalbe much more musicalbe free of speech and hearing problemsbe free of speech and hearing problemsbe able to learn any language without accentbe able to learn any language without accentsee things we hardly noticesee things we hardly notice......

Popular book: The Scientist in the Crib, Popular book: The Scientist in the Crib, Alison Gopnik, Andrew N. Meltzoff, Patricia K. Kuhl (William Morrow & Alison Gopnik, Andrew N. Meltzoff, Patricia K. Kuhl (William Morrow & Company 1999.Company 1999.