ECE 599/692 – Deep Learning

Lecture 19 – Beyond BP and CNNHairong Qi, Gonzalez Family ProfessorElectrical Engineering and Computer ScienceUniversity of Tennessee, Knoxvillehttp://www.eecs.utk.edu/faculty/qiEmail: hqi@utk.edu

1

Schedule

• Nov. 9: Beyond BP and CNN• Nov. 14: Depth vs. Breadth

– What’s beyond ReLU?– What’s beyond cross entropy?

• Nov. 16: Projects discussion– What’s beyond leaderboard?

• Nov. 21: How to design a new network structure?• Nov. 28, 30, Dec. 5

– Final project presentation (26 presentations)

2

From Big Data to Artificial Intelligence• What is big data?• The more the better?• Can’t live without data?• AlphaGo Zero vs. AlphaGo

– No human input vs. Human knowledge• Hinton’s two new papers on capsule networks (2017)

– “I think the way we’re doing computer vision is just wrong… It works better than anything else at present but that doesn’t mean it’s right.”

– Capsule (small groups of crude virtual neurons) networks for tracking different parts of an object

– Abandon BP– “…humans should encode as little knowledge as possible into AI

software, and instead make them figure things out for themselves from scratch…”

– “The future of AI is determined by those graduate students who seriously doubt all what I have said.”

3https://deepmind.com/blog/alphago-zero-learning-scratch/

A bit history• 1943 (McCulloch and Pitts):• 1957 - 1962 (Rosenblatt):

– From Mark I Perceptron to the Tobermory Perceptron to Perceptron Computer Simulations– Multilayer perceptron with fixed threshold

• 1969 (Minsky and Papert):• The dark age: 70’s ~25 years• 1986 (Rumelhart, Hinton, McClelland): BP• 1989 (LeCun et al.): CNN (LeNet)• Another ~20 years• 2006 (Hinton et al.): DL• 2012 (Krizhevsky, Sutskever, Hinton): AlexNet• 2014 (Goodfellow, Benjo, et al.): GAN

4

• W.S. McCulloch, W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The Bulletin of Mathematical Biophysics, 5(4):115-133, December 1943.

• F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan Books, 1962.• Minsky, S. Papert, Perceptrons: An Introduction to Computational Geometry, 1969.• D.E. Rumelhart, G.E. Hinton, R.J. Williams, “Learning representations by back-propagating errors,” Nature, 323(9):533-536, October 1986.

(BP)• Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, “Backpropagation applied to handwritten zip code

recognition,” Neural Computation, 1(4):541-551, 1989. (LeNet).• G.E. Hinton, S. Osindero, Y. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, 18:1527-1554, 2006. (DL)• G.E. Hinton, R.R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, 313(5786):504-507, 2006 (DL)• A. Krizhevsky, I. Sutskever, G.E. Hinton, “ImageNet classification with deep convolutional neural networks,” Advances in Neural Information

Processing Systems, pages 1097-1105, 2012. (AlexNet)• I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, “Generative adversarial networks,” NIPS,

2014.

A bit of history - revisited• 1956-1976

– 1956, The Dartmouth Summer Research Project on Artificial Intelligence, organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon

– The rise of symbolic methods, systems focused on limited domains, deductive vs. inductive systems

– 1973, the Lighthill report by James Lighthill, “Artificial Intelligence: A General Survey” -automata, robotics, neural network

– 1976, the AI Winter• 1976-2006

– 1986, BP algorithm– ~1995, The Fifth Generation Computer

• 2006-???– 2006, Hinton (U. of Toronto), Bingio (U. of Montreal, LeCun (NYU)– 2012, ImageNet by Fei-Fei Li (2010-2017) and AlexNet

5

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College ... The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

https://en.wikipedia.org/wiki/Dartmouth_workshophttps://en.wikipedia.org/wiki/Lighthill_report

ML vs. AI

• ML + inference à AI

6

Turing awardees in AI

– 1969, Marvin Minsky– 1971, John McCarthy– 1975, Allen Newell and Herbert A. Simon– 1994, Edward Feigenbaum and Raj Reddy– 2010, Leslie G. Valiant– 2011, Judea Pearl

7

https://en.wikipedia.org/wiki/Turing_Award

Artificial Intelligence• The three branches

– Logic-based (Symbolic method)– Network-based– Behavior-based (Self adaptive and evolution)

• Artificial intelligence vs. Human intelligence– Logical– Linguistic– Spatial– Musical– Kinesthetic– Intra-personal– Inter-personal– Naturalist– Graphics

8

Government involvement

• China’s new generation artificial intelligence– Big data intelligence– Swarm intelligence– Multimedia (Cross-domain) intelligence (speech, image, text,

natural language)– Human-machine coordination– Autonomous vehicle

• Applications– Computer vision– Speech recognition– Natural language processing– Human-machine interface– Robotics

9

Andrew Parker’s Light Switch Theory• Andrew Parker, In The Blink of An Eye: How

Vision Sparked The Big Bang Of Evolution, Basic Books, 2004.

• The Cambrian explosion: the explosion of life forms (550 million years ago)

• The theory:– It was the development of vision in primitive animals

that caused the explosion.

10

https://www.amazon.com/Blink-Eye-Vision-Sparked-Evolution/dp/0465054382

Visual intelligence: Beyond ImageNet• Fei-Fei Li’s talk at CNCC 2017• Object recognition (ImageNet – Single Object

Recognition)– 0.28 (2010) à 0.26 (2011) à 0.16 (2012) à 0.12 (2013) à 0.07

(2014) à 0.036 (2015) à 0.03 (2016) à 0.023 (2017)• Beyond object recognition à Rich scene gist

– The Visual Genome Dataset – Visual relationship à Semantic scene retrieval à Scene graph

generation• Beyond scene gist à Vision + Language & Reasoning

– The CLEVR Dataset– Image captioning à Dense Captioning à Paragraph generation

• https://www.leiphone.com/news/201710/CvdrhzTO0cndEArJ.html

11

What’s after ImageNet?

• MS COCO (Microsoft Common Objects in Context)– 4 Tasks: Detection challenge, Instance segmentation, Human

keypoint challenge, Stuff segmentation– Winners (Detection): MSRA (2015), Google (2016), Face++

(2017)– http://cocodataset.org/#home

• Places (MIT and CMU)– 3 Tasks: Scene parsing, Instance segmentation, and Boundary

detection– Winners (2017):

– Scene parsing: CAS– Instance segmentation: Face++

– http://placeschallenge.csail.mit.edu/results_challenge.html

12

13

Reference

• Fei-Fei Li, Visual Intelligence Beyond ImageNet, CNCC 2017,https://www.leiphone.com/news/201710/CvdrhzTO0cndEArJ.html

• Xiangyang Shen, Microsoft, CNCC 2017• Wen Gao, From Big Data to Artificial Intelligence,

JDDiscovery, 2017,https://www.toutiao.com/a6485167794265522701/?tt_from=weixin&utm_campaign=client_share&app=news_article&utm_source=weixin&iid=9368489672&utm_medium=toutiao_android&wxshare_count=1

14