10th kovacs colloquium unesco water resource planning and management using motivated machine...

10th Kovacs Colloquium UNESCO

Water Resource Planning and Water Resource Planning and Management using Motivated Management using Motivated Machine LearningMachine Learning

Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA

www.ent.ohiou.edu/~starzyk

10th Kovacs Colloquium UNESCO, Hydrocomplexity:

New Tools for Solving Wicked Water


Challenges in Water Management Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Motivated Learning ML Experiment Abstract Motivations and Goal

Hierarchy Promises of EI

OutlineOutline

Flood in Poland


Water management is challenging since:

Strategies are developed mostly on national level

There is a competition between countries for water

Water policy plans effects environment and society, health and development, and economy

Growing demands for water Need to integrate water management and

policy making There is an acute need for legitimate

scientific data Decision making in water-related health, food

and energy systems are critical to economical development and national security

Challenges in Water Challenges in Water ManagementManagement

South–North Water Transfer Project China


Decision makers must consider important questions:

How do we make water use sustainable? How to protect water resources from overuse

and contamination? Water problems are interconnected and too

complex to be handled by a single institution or a single group of people

It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use

It is necessary to create assessment and modeling tools to improve policy making resolve conflicting issues and facilitate interaction.



Why accurate integrated models to support decision making are important ?

Computerized models were used for many years to support water related decisions.

Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions.

This work proposes models that emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies.

The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making.



What are deficiencies of machine created models?

Artificial neural networks, fuzzy logic, and genetic algorithms have been used to model resource planning and water management

It is difficult to apply these tools in real-life decisions as the related parameters are not explicitly known

This work presents a machine learning approach that motivates machine to develop into a useful toll.

Motivated machine learning can characterize data and make predictions about their dynamic changes It could support intelligent decision making in

dynamically changing environment It could observe impacts of alternative water

management policies It may consider social, cultural, political, economic and

institutional elements of decision making



Embodied intelligence may support decision making:

EI mimics biological intelligent systems, extracting general principles of intelligent behaviour

It uses emerging, self-organizing, goal creation (GC) system that motivates EI to learn how to interact with the environment Knowledge is not entered into such systems, but is a

result of useful actions in a given environment. This knowledge is validated through active interaction

with the environment. Motivated intelligent systems adapt to unpredictable

and dynamic situations in the environment by learning, which gives them a high degree of autonomy

Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory



How to use the motivated learning scheme to integrate modelling and decision making?

It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty. For instance, using classical machine learning to

represent physical processes works only under the assumption that the same processes will repeat.

However, if a process changes beyond certain limits, the prediction will not be correct.

ML systems may overcome this difficulty and such systems can be trained to advice humans.

Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk

The talk will explain internal motivation mechanism that leads to effective goal oriented learning, abstract goal creation and goal management



IntelligenceIntelligence

http://www.home-business-smarts.net/

Mainstream Science on Intelligence December 13, 1994: An Editorial With 52 Signatories, by Linda S. Gottfredson, University of Delaware

Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.


Animals’ Intelligence Animals’ Intelligence Defining intelligence

through humans is not appropriate to design intelligent machines:

– Animals are intelligent too

Dog IQ test: Dogs can learn 165 words (similar to 2 year olds) Average dog has the mental abilities of a 2-year-old child (or better) They would beat a 3- or 4-year-old in basic arithmetic, Dogs show some basic emotions, such as happiness, anger and disgust “The social life of dogs is very complex - more like human teenagers -

interested in who is moving up in the pack, who is sleeping with who etc,“ says professor Stanleay Coren from University of British Columbia


Computational Models of Intelligence Computational Models of Intelligence

Five paradigms of Computational intelligence

http://rtpis.mst.edu/images/paradigms_of_CI.jpg

How to define and compute intelligence?


Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence

attempt to simulate “highest” human faculties:

– language, discursive reason, mathematics, abstract problem solving

Environment model Condition for problem

solving in abstract way “brain in a vat”

Embodiment knowledge is implicit in the

fact that we have a body– embodiment supports brain

development

Intelligence develops through interaction with environment Situated in environment Environment is its best model


Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999

Interaction with complex environment

cheap design ecological balance redundancy principle parallel, loosely

coupled processes asynchronous sensory-motor

coordination value principle

Agent

Drawing by Ciarán O’Leary- Dublin Institute of Technology


Embodied Intelligence Embodied Intelligence

– Mechanism: biological, mechanical or virtual agent

with embodied sensors and actuators– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI

Definition Embodied Intelligence (EI) is a

mechanism that learns how to survive in a hostile environment


Embodied Intelligence Embodied Intelligence

For water resource planning and management hostility of the environment means Insufficient water resources Poor water quality Growing demand of industry for water Conflicts between stakeholders, etc

These hostile signals represent the primitive pains that grow unless they are addressed by proper actions

Surviving in this environment (politically) is to keep these signals below specified level, otherwise economical crises, social unrest, drought or famine will follow


Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment of a MindEmbodiment of a Mind

Embodiment is a part of the environment that EI controls to interact with the rest of the environment It contains intelligence core

and sensory motor interfaces under its control

Necessary for development of intelligence

Not necessarily constant or in the form of a physical body

Boundary transforms modifying brain’s self-determination


Brain learns own body’s dynamic Self-awareness is a result of

identification with own embodiment Embodiment can be extended by

using tools and machines Successful operation is a function

of correct perception of environment and own embodiment

Embodiment of a MindEmbodiment of a Mind


INPUT OUTPUT

Simulation or

Real-World System

TaskEnvironment

Agent Architecture

Long-term Memory

Short-term Memory

Reason

ActPerceive

RETRIEVAL LEARNING

EI Interaction with EnvironmentEI Interaction with Environment

From Randolph M. Jones, P : www.soartech.com


How to Motivate a MachineHow to Motivate a Machine ? ?

The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity?

How to motivate it to explore the environment and learn how to effectively work in this environment?

Can a machine that only implements externally given goals be intelligent?If not how these goals can be created?


I suggest that hostility of environment motivates us. It is the pain that moves us. Our intelligence that tries to minimize this pain motivates our actions,

learning and development

We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain

How to Motivate a MachineHow to Motivate a Machine ? ?

In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop.

Without the pain there will be no motivation to develop.


Motivated Learning Motivated Learning I suggest a goal-driven mechanism to motivate

a machine to act, learn, and develop. A simple pain based goal creation system. It uses externally defined pain signals that are

associated with primitive pains. Machine is rewarded for minimizing the primitive

pain signals.

Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent. Machine creates abstract goals based on the primitive pain signals. It receives internal rewards for satisfying its goals (both primitive and

abstract). ML applies to EI working in a hostile environment.


Pain-center and Goal CreationPain-center and Goal Creation Simple Mechanism Creates hierarchy of

values Motivation is to reduce

the primitive pain level Leads to formulation of

complex goals Reinforcement :

• Pain increase• Pain decrease

Forces exploration

+

-

Environment

Sensor

MotorPain level

Dual pain levelPain increase

Pain decrease

(-)

(+)

Motivation

(-)

(-)

(+)

(+)

Wall-E’s goal is to keep his plants from dying

(+)

(-)

Goal


Primitive Goal CreationPrimitive Goal Creation

- +

Pain

Dry soilPrimitive

level

opentank

sit on garbage

refillfaucet

w. can water

Dual pain

Reinforcing a proper action


Abstract Goal HierarchyAbstract Goal Hierarchy

Abstract goals are created to reduce abstract pains in order to satisfy the primitive goalsA hierarchy of abstract goals is created - they satisfy the lower level goals

ActivationStimulationInhibitionReinforcementEchoNeedExpectation

- +

+

Dry soilPrimitive Level

Level I

Level IIfaucet

-

w. can

open

water

+

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Level IIItank

-

refill


Motivated Learning ExperimentMotivated Learning Experiment

Sensory-motor pairs and their effect on the environment

Sensory Motor Increases Decreases

Dry soil Water from Can Moisture Water in Can

No Water in Can Water from Tank Water in Can Water in Tank

No Water in Tank Water from Reservoir Water in Tank Water in Reservoir

No Water in Reservoir Water from Lake Water in Reservoir Water in Lake

No Water in Lake Regulate Usage Water in Lake -

Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”


Action scatters in 5 ML simulations

0 100 200 300 400 500 6000

5

10

15

20

25

30

35

40Goal Scatter Plot

Go

al I

D

Discrete time



The average pain signals in 100 ML simulations

0 100 200 300 400 500 6000

0.5

Primitive pain – dry soil

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in can

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in tank

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in reservoir

Pa

in

0 100 200 300 400 500 6000

0.050.1

Lack of water in lake

Pa

in

Discrete time



Averaged performance over 10 trials:

MLRL

Machine using ML learns to control all abstract pains and maintains the primitive pain signal on a low level.

ML vs. Reinforcement LearningML vs. Reinforcement Learning


Multiple dependencies:- two resources that can provide money

Motivated Learning Experiment IIMotivated Learning Experiment II


When the environment is abundant in both resources

Competition between Work and Sell valuables

The loser and its associated further goals will be ignored by the system

Motivated Learning Experiment IIMotivated Learning Experiment II


ML Abstract Goal HierarchyML Abstract Goal Hierarchy


Compare ML and RLCompare ML and RLMean primitive pain Pp value as a function of the number of iterations.

>10 levels of hierarchy >complex environment

- green line for RL - blue line for ML.


Reinforcement LearningReinforcement Learning Motivated Learning Motivated Learning Single value function Measurable rewards

Can be optimized

Predictable Objectives set by

designer Maximizes the reward

Potentially unstable

Learning effort increases with complexity

Always active

Multiple value functions One for each goal

Internal rewards Cannot be optimized

Unpredictable Sets its own objectives Solves minimax problem

Always stable

Learns better in complex environment than RL

Acts when needed

http://www.bradfordvts.co.uk/images/goal.jpg


Machine Working for Humanity?Machine Working for Humanity? If you’re trying to look far ahead, and

what you see seems like science fiction, it might be wrong.

But if it doesn’t seem like science fiction, it’s definitely wrong.

From presentation by Foresight Institute


Questions?Questions?

10th kovacs colloquium unesco water resource planning and management using motivated machine...

Documents

water management slide

acceptable water

water resources

water problems

wicked water slide

water related decisions

waterrelated health

management decisions