Artificial IntelligenceAcademic year 2016/2017

Giorgio Fumerahttp://pralab.diee.unica.it

fumera@diee.unica.it

Pattern Recognition and Applications LabDepartment of Electrical and Electronic Engineering

University of Cagliari

Outline

Part I: Historical notes

Part II: Solving Problems by SearchingUninformed SearchInformed search

Part III: Knowledge-based SystemsLogical LanguagesExpert systems

Part IV: The Lisp Language

Part V: Machine LearningDecision TreesNeural Networks

Part I

Historical Notes

Artificial Intelligence

A discipline born in the 1950s.Goal: building “intelligent” machines.A goal envisaged since a long time before, e.g.:

I Leonardo da Vinci’s robotknight (about 1495)

I Automaton chess player(“Mechanical Turk”, late18th century)

I science fiction

What is “intelligence”?

Broad definition: a set of capabilities that allow humans toI learnI thinkI understandI communicateI be self-consciousI build abstract models of the worldI planI adapt to novel external conditionsI . . .

Some of these capabilities are exhibited also by animals (e.g.,associative memory, reacting to stimuli, communicating).

Early investigations: clues from humans

Goal: understanding human intelligence.I High-level manifestation: rationality

Logic: Aristotle (4th cent. bc), G.W. Leibniz (17th-18thcent.), G. Boole (19th cent.), etc.

I High-level manifestation: behavior and mindPsychology and cognitive science (since 19th cent.)

I Low-level biological support: the brainNeuroanatomy and Neurophysiology (since 19th cent.):McCulloch and Pitt’s model of neuron (1943), D.O. Hebb’stheory on neurons as basic units of thought, etc.

Early investigations: clues from technology

Goal: building machines capable of autonomously performing sometask.

I Automata (e.g., jacquard loom, 1804)Cybernetics (feedback and control):N. Wiener, W.R. Ashby (1940s)

I Statistics and probability as tools to dealwith uncertainty in reasoning anddecision-making: T. Bayes (18th cent.);K. Pearson, R.A. Fisher, A. Wald,J. Neyman (late 19th – 20th cent.)

Early investigations: clues from technology

The computer:I precursors: mechanical devices: B. Pascal and G.W. Leibniz (17th

cent.), C. Babbage’s analytical engine (19th cent.)

I from engineering: electromechanical and electronic devices(1940s): K. Zuse, J.P. Eckert, J.W. Mauchly, J. von Neumann

I from logic and mathematics: computational theory, the foundationof computer science: A.M. Turing, A. Church (1930s)

Early investigations: “thinking” computers

Electronic computers: the right tool for building intelligentmachines?

Alan M. Turing’s (1912-1954) main contributions:

I first investigations into the nature ofcomputing

I the “logical computing machine” (Turingmachine): a universal computer

I envisioning intelligent computers:Computing Machinery and Intelligence,Mind, Vol. LIX, No. 263, 433–460, 1950;operational definition of intelligence: theTuring Test

The birth of AI

1950s: many researchers from di�erent disciplines investigateintelligence under two main viewpoints:

I understanding human intelligenceI building intelligent machines

Dartmouth College Workshop (USA, summer 1956): o�cial birthof AI, as a discipline aimed at building intelligent machines.The founders: J. McCarthy, M. Minsky, A. Newell, H. Simon,C. Shannon, O. Selfridge, R. Solomono�, . . .

Early explorations: 1950s and 1960s

Goals: identifying specific tasks that require intelligence, andfiguring out how to get machines to do them.

Great interest in mimicking high-level human thought and mentalabilities, e.g.:

I reasoningI understanding natural languageI understanding images

and some related low-level abilities:I recognizing speech soundsI distinguishing objects in imagesI reading cursive script

But: how do humans do that?

Early explorations: 1950s and 1960s

Starting point: toy problems, and some real-world onesI game playing 15-puzzle, checkers, chess, etc.I theorem provingI natural language processing (NLP)I recognizing objects in images

Early explorations: 1950s and 1960s

First approaches and tools:I symbolic (high-level) approach: based on the assumption

that the essence of intelligence is symbol processing– heuristic search– syntax analysis/generation– symbolic knowledge representation (symbols, lists)– symbolic knowledge processing: new programming languages

(LISP, etc.)I non-symbolic (low-level) approach

– pattern recognition– artificial neural networks

Heuristic search methodsSymbol processing approach, applied to:

I game playing: 15-puzzle, checkers, chess (the “Drosophila of AI”)

I geometric analogy problems

I theorem provingI mechanizing problem solving: A. Newell and H. Simon’s General

Problem Solver (1959)

Heuristic search methods

Common approach:I knowledge representation: lists of symbols

(main feature of the LISP language, 1958–, J. McCarthy)I search methods: search tree, heuristics

Natural language processing

Symbol processing approach.Goals: understanding, generating, translating natural language.Linguistic levels:

I morphology: word parts (e.g.: walking = walk + -ing)I syntax (grammar): rules that define well-formed sentences

(e.g.: John hit the ball : Yes; ball the hit John: No)I semantics: meaning of a sentenceI pragmatics: context and background knowledge, e.g.:

John went to the bankJohn threw the ball to the window and broke itJohn threw the glass to the wall and broke it

Natural language processing

Main focus of early research: syntactic level.Seminal work: N. Chomsky, Syntactic Structures, 1957.Grammar definition: syntax rules for analyzing/generatingsentences; main tool: parse tree.

Applications:I question answering (original goal: computer interfaces)I machine translation: early optimism, but a very hard task

Pattern recognition

Goal: classifying input signals (images, sounds, electronic signals,etc.) into one of several categories.First problems: image classification.Early e�orts: optical character recognition (OCR).

Main approaches:I template matchingI learning

1. image pre-processing (noise filtering, line thickening, edgeenhancement, ...)

2. feature extraction3. classification “rules” learnt from examples

Artificial neural networks

Non-symbolic (low-level), connectionist approach.F. Rosenblatt (psychologist): perceptron (1957–) as a potentialmodel of human learning, cognition and memory:

I network of McCulloch-Pitts’ neural elementsI learning algorithm for adjusting connection weights from

examplesFirst applications: pattern recognition

I OCRI aerial image recognition

Great expansion: mid–1960s to early 1980s

From toy/lab problems to real-world and commercial applications:I computer visionI mobile robotsI game playingI speech recognition, NLPI knowledge representation and reasoning

Research funding:I DARPA’s Strategic Computing Program (USA)I Fifth Generation Computer Systems (Japan)I ESPRIT (Europe)

Great expansion: mid–1960s to early 1980sComputer vision:

I Summer Vision project (MIT, 1966)I low-level, hierarchical image processing (hints from biology); image

filters; line, corner, surface detection; 3D reconstructionI early application: guiding a robot arm to manipulate blocks

I high-level vision: finding objects in scenes (templates, parts)I two main approaches emerge:

– whole scene reconstruction: di�cult– perceiving to guide robot action (purposive vision): easier

Great expansion: mid–1960s to early 1980s

Mobile robots:I sensors, actuators, computer vision, environment modeling,

planningI route finding: heuristic search, A* algorithmI first autonomous vehicles

Great expansion: mid–1960s to early 1980s

Game playing: progress in chessI programs that attain human-level (not master) capabilityI investigations into human ability: accumulated knowledge vs

massive search

Great expansion: mid–1960s to early 1980sNLP: greater success on less ambitious goals

I improvements in grammarsI machine translation with humans in the loopI declarative (logical languages, inference algorithms) vs procedural

(“hard-wired”) knowledgeI dialog systems (1971, T. Winograd’s SHRDLU: blocks’ world,

procedural knowledge)

I speech recognition (easy), and understanding (di�cult)

Great expansion: mid–1960s to early 1980s

Knowledge representation and reasoning:I consulting / decision support / expert systems

main idea: solving domain-specific problems by embeddingexpert knowledge in the form of IF-THEN rules

I applications: chemistry, medical diagnosis, geology, military;since 1990s: business

Great expansion: mid–1960s to early 1980s

Summing up, until the 70s AI research is mainly based on thesymbol processing conception of human intelligence

I main approach to AI: mimicking high-level human abilitiesthrough heuristic search and symbolic processing (“good,old-fashioned AI”, GOFAI)

I many successful applications through a pragmatic approach inspecific tasks . . .

I . . . but very limited achievements with respect to earlyexpectations for a “general” AI

Mid 1980s: the “AI winter”

Real-world tasks require much more “intelligence” than thatachievable by heuristic search and symbolic processing (GOFAI).Two main issues emerge:

I computational complexity: combinatorial explosionI need of a large body of knowledge (including common sense)

Also the non-symbolic, connectionist approach (artificial neuralnetworks) exhibits limitations.

Main consequences:I drop of interest in AII scaling back AI’s goalsI cut of research funds

Mid–1980s to 1990s: technical and theoretical advances

New results in several fields, solid theoretical foundations from:I mathematicsI statistics and probability theoryI control engineering

Concrete progress in real-world tasks (albeit still far from initialexpectations).

Mid–1980s to 1990s: technical and theoretical advances

Knowledge representation and reasoning:I new paradigms, e.g.; fuzzy logic, soft computing (inspired by

human mind)I semantic networks, ontologies; e.g.:

– WordNet, http://wordnet.princeton.edu– BabelNet, http://babelnet.org

I probabilistic reasoning to overcome the limits of logic(probabilistic graphical models, Bayesian networks, learning)

Search algorithms: evolutionary approach, genetic algorithms(inspired by evolution).

Mid–1980s to 1990s: technical and theoretical advances

The rise of machine learning:I huge amount of data in digital form become availableI main idea: automatically inferring knowledge (patterns, rules,

etc.) from data instead of eliciting it from domain expertsI data analysis methods: data mining, etc.I theoretical foundations: statisticsI novel techniques: inductive logic programming, decision trees,

resurgence of ANNs (1986: back-propagation algorithm),support vector machines, ensemble methods, etc.

I applications to many fields: computer vision, natural languageprocessing, etc.

Mid–1980s to 1990s: technical and theoretical advances

Computer vision:I two main approaches persist: scene analysis, purposive visionI main achievements: surface, depth; tracking, object

recognitionI fruitful exchanges with research on animal/human visionI novel techniques: hierarchical models, ANNs, deep neural

networksI extensive application of machine learning techniques

Mid–1980s to 1990s: technical and theoretical advances

Intelligent Agent architectures:I sensor networksI autonomous, cooperating robots; emergent behaviorI the intelligent agent paradigm

A toy (?) example: soccer-playing robots

Mid 1990s – today: main achievementsOriginal goal (building intelligent machines): still far-reaching.But: successes in real-world problems, many commercialapplications, many startup companies.Examples:

I games: master-level programs in checkers, chess and (veryrecently) Go

I robots: driverless automobiles, space vehiclesI pervasive applications: home, cars, route finding in maps

(search algorithms), recommender systems (machine learning,social/collaborative filtering), characters in video games, ...

I automated (high-frequency) tradingI medicineI business rule management systemsI translating systemsI computer vision (face recognition, ...)

Summary of the main approaches to AI

Four distinct approaches to build intelligent machines:

Human performance RationalityMind/ Systems that think Systems that thinkthinking like humans rationally

(cognitive modeling approach) (“law of thought” approach)

BehaviorSystems that act Systems that actlike humans rationally(Turing test approach) (rational agent approach)

Rational agents (acting rationally): the most general approach,amenable to scientific/technological development (though, maybe,not useful enough for understanding human intelligence)

A snapshot of current AI research

Associations:I Association for the Advancement of Artificial Intelligence (AAAI)

www.aaai.org

I European Coordinating Committee for Artificial Intelligence(ECCAI), www.eccai.org

I Italian Association for Artificial Intelligence (AI*IA)www.aixia.it

Conferences:I Int. Joint Conf. on Artificial Intelligence, ijcai.org

Scientific journals:I Artificial Intelligence

www.journals.elsevier.com/artificial-intelligence

I J. of Artificial Intelligence Research, www.jair.org

A snapshot of current AI research

Research topics:I Machine learningI Knowledge representation and reasoningI Reasoning under uncertainty or imprecisionI Natural languageI VisionI Pattern recognitionI Heuristic searchI Intelligent roboticsI Multi-agent systemsI PlanningI . . .

Philosophical issues

A long-standing question: Can machines be intelligent?Two main hypotheses:

I Weak AI: machines can emulate intelligence (act intelligently)I Strong AI: machines can be intelligent

(if they act intelligently, they are intelligent, e.g.: Turing test)

Another long-standing question: Is (human) mind a machine?

Philosophical issues

Some arguments raised against Weak AI:I machines can never do . . . (make mistakes, learn from

experience, have a sense of humor, enjoy ice creams, . . . )I machines are formal systems, and formal systems cannot

establish the truth of every mathematical sentence (Gödelincompleteness theorem), whereas humans (in principle) can

I human behavior can not be captured by a set of rulesA. Turing’s viewpoint (Mind, 1950):Can a machine think? is an ill-posed question.Consider this one: Can a machine fly/swim?

I airplanes “fly”, but not as birds “fly”I ships “swim” in Russian, not in English or in Italian...

Philosophical issues

Some issues about Strong AI:I even if machines can emulate intelligence, they cannot be

self-consciousI relationship between mental states and body (brain) states

(free will, consciousness, intentions): dualism (R. Descartes,17th cent.) vs materialism (“brains cause mind”)

I a machine running the “right” program (e.g., for naturallanguage understanding) does not necessarily have a mind(the Chinese room thought experiment, J. Searle, 1980)

I intelligence is an emerging behavior that can be onlysupported by biological brains

Ethical issues

I Even if we could build intelligent machines, should we?I Consequences on humans: loss of jobs, loss of the sense of

being unique, end of human race, etc.I Accountability (e.g., driverless cars)I . . .

Ethical concerns are now re-flourishing, as many believe thathuman-level AI is now in reach (e.g., current focus of the Future ofLife Institute, http://futureoflife.org).

Some recent projects

Human Brain Project (EU)https://www.humanbrainproject.euOverall goal: understanding the human brain and itsdiseases, and emulating its computational capabilities

RoboLaw (EU)http://www.robolaw.euRegulating Emerging Robotic Technologies in Europe:Robotics facing Law and Ethics