alphago for disco group of eth zurich

AlphaGo: Mastering the game of Go

with deep neural networks and tree search

Karel Ha

article by Google DeepMind

ETH Zurich, 9th May 2016

Game AIs

https://xkcd.com/1002/ 0

Heads-up Limit Holdem Poker Is Solved!

Cepheus http://poker.srv.ualberta.ca/

0.000986 big blinds per game on expectation

Bowling et al. 2015 1

Game Tree

Tree Search

Optimal value v∗(s) determines the outcome of the game:

� from every board position or state s

� under perfect play by all players.

It is computed by recursively traversing a search tree containing

approximately bd possible sequences of moves, where

� b is the games breadth (number of legal moves per position)

� d is its depth (game length).

Silver et al. 2016 2

Tree Search

Game tree of Go

Sizes of trees for various games:

� chess: b ≈ 35, d ≈ 80

� Go: b ≈ 250, d ≈ 150

⇒ more positions than atoms in the

universe!

That makes Go a googol

[10100] times more complex

than chess.

https://deepmind.com/alpha-go.htmlHow to handle the size of the game tree?

� for the breadth: a neural network to select moves

� for the depth: a neural network to evaluate the current

position

� for the tree traverse: Monte Carlo tree search (MCTS)

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

� Go: b ≈ 250, d ≈ 150 ⇒ more positions than atoms in the

universe!

than chess.

position

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

universe!

than chess.

https://deepmind.com/alpha-go.html

How to handle the size of the game tree?

position

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

universe!

than chess.

position

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

universe!

than chess.

position

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

universe!

than chess.

position

Allis et al. 1994 3

Game tree of Go

� chess: b ≈ 35, d ≈ 80

universe!

than chess.

position

� for the tree traverse: Monte Carlo tree search (MCTS)Allis et al. 1994 3

AlphaGo: Neural Networks

Policy and Value Networks

Training the (Deep Convolutional) Neural Networks

SL (Supervised Learning) Policy Network

� 13-layer deep convolutional neural network

� goal: to predict expert human moves

� task of classification

� trained from 30 millions positions from the KGS Go Server

� stochastic gradient ascent:

∆σ ∝ ∂ log pσ(a|s)

(to maximize the likelihood of the human move a selected in state s)

Results:

� 44.4% accuracy (the state-of-the-art from other groups)

� 55.7% accuracy (raw board position + move history as input)

� 57.0% accuracy (all input features)

Results:

Rollout Policy

� Rollout policy pπ(a|s) is faster but less accurate than SL

policy network.

� accuracy of 24.2%

� It takes 2µs to select an action, compared to 3 ms in case

of SL policy network.

Rollout Policy

policy network.

Rollout Policy

policy network.

RL (Reinforcement Learning) Policy Network (1/2)

� identical in structure to the SL policy network

� goal: to win in the games of self-play

� weights ρ initialized to the same values, ρ := σ

� games of self-play

� between the current RL policy network and a randomly

selected previous iteration

� to prevent overfitting to the current policy

∆ρ ∝ ∂ log pρ(at |st)

∂ρzt

at time step t, where reward function zt is +1 for winning and −1 for losing.

∂ρzt

Results (retrieved by sampling each move at ∼ pρ(·|st)):

� 80% of win rate against the SL policy network

� 85% of win rate against the strongest open-source Goprogram, Pachi (Baudis and Gailly 2011)

� The previous state-of-the-art, based only on SL of CNN:

11% of “win” rate against Pachi

Value Network (1/2)

� similar architecture to the policy network, but outputs a single

prediction instead of a probability distribution

� goal: to estimate a value function

vp(s) = E[zt |st = s, at...T ∼ p]

that predicts the outcome from position s (of games played

by using policy p)

� Double approximation: vθ(s) ≈ vpρ(s) ≈ v∗(s).

� task of regression

� stochastic gradient descent:

∆θ ∝ ∂vθ(s)

∂θ(z − vθ(s))

(to minimize the mean squared error (MSE) between the predicted vθ(s) and the true z)

Value Network (1/2)

vp(s) = E[zt |st = s, at...T ∼ p]

by using policy p)

∆θ ∝ ∂vθ(s)

∂θ(z − vθ(s))

Value Network (1/2)

vp(s) = E[zt |st = s, at...T ∼ p]

by using policy p)

∆θ ∝ ∂vθ(s)

∂θ(z − vθ(s))

Value Network (1/2)

vp(s) = E[zt |st = s, at...T ∼ p]

by using policy p)

∆θ ∝ ∂vθ(s)

∂θ(z − vθ(s))

Value Network (1/2)

vp(s) = E[zt |st = s, at...T ∼ p]

by using policy p)

∆θ ∝ ∂vθ(s)

∂θ(z − vθ(s))

Value Network (2/2)

Beware of overfitting!

� Consecutive positions are strongly correlated.

� Value network memorized the game outcomes, rather than

generalizing to new positions.

� Solution: generate 30 million (new) positions, each sampled

from a seperate game

� almost the accuracy of Monte Carlo rollouts (using pρ), but

15000 times less computation!

Value Network (2/2)

Elo Ratings for Various Combinations of Networks

AlphaGo: The Main Algorithm

MCTS Algorithm

The next action is selected by lookahead search, using simulation:

1. selection phase

2. expansion phase

3. evaluation phase

4. backup phase (at end of all simulations)

Each edge (s, a) keeps:

� action value Q(s, a)

� visit count N(s, a)

� prior probability P(s, a) (from SL policy network pσ)

The tree is traversed by simulation (descending the tree) from the

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm

1. selection phase

2. expansion phase

3. evaluation phase

root state.

MCTS Algorithm: Selection

At each time step t, an action at is selected from state st

at = arg maxa

(Q(st , a) + u(st , a))

where bonus

u(st , a) ∝P(s, a)

1 + N(s, a)

at = arg maxa

(Q(st , a) + u(st , a))

where bonus

u(st , a) ∝P(s, a)

1 + N(s, a)

at = arg maxa

(Q(st , a) + u(st , a))

where bonus

u(st , a) ∝P(s, a)

1 + N(s, a)

MCTS Algorithm: Expansion

A leaf position may be expanded (just once) by the SL policy network pσ .

The output probabilities are stored as priors P(s, a) := pσ(a|s).

MCTS: Evaluation

� evaluation from the value network vθ(s)

� evaluation by the outcome z using the fast rollout policy pπ until the end of game

Using a mixing parameter λ, the final leaf evaluation V (s) is

V (s) = (1− λ)vθ(s) + λz

MCTS: Evaluation

V (s) = (1− λ)vθ(s) + λz

MCTS: Evaluation

V (s) = (1− λ)vθ(s) + λz

MCTS: Evaluation

V (s) = (1− λ)vθ(s) + λz

MCTS: Evaluation

V (s) = (1− λ)vθ(s) + λz

MCTS: Backup

At the end of simulation, each traversed edge is updated by accumulating:

� the action values Q

� visit counts N

MCTS: Backup

At the end of simulation, each traversed edge is updated by accumulating:

� the action values Q

� visit counts N

Once the search is complete, the algorithm

chooses the most visited move from the root

position.

Principal Variation (Path with Maximum Visit Count)

The moves are presented in a numbered sequence.

� AlphaGo selected the move indicated by the red circle;

� Fan Hui responded with the move indicated by the white square;

� in his post-game commentary, he preferred the move (labelled 1) predicted by AlphaGo.

AlphaGo: Distributed Computing

Scalability

� asynchronous multi-threaded search

� simulations on CPUs

� computation of neural networks on GPUs

AlphaGo (on a single-machine):

� 40 search threads

� 40 CPUs

� 8 GPUs

Distributed version of AlphaGo (on multiple machines):

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Scalability

� 40 CPUs

� 8 GPUs

� 1202 CPUs

� 176 GPUs

Elo Ratings for Various Combinations of Threads

AlphaGo: Results

http://xkcd.com/1263/ 23

Tournament with Other Go Programs

Fan Hui

� professional 2 dan

� European Go Champion in 2013, 2014 and 2015

� European Professional Go Champion in 2016� biological neural network:

� 100 billion neurons

� 100 up to 1,000 trillion neuronal connections

https://en.wikipedia.org/wiki/Fan_Hui 25

Fan Hui

� European Professional Go Champion in 2016

� biological neural network:

Fan Hui

� 100 up to 1,000 trillion neuronal connectionshttps://en.wikipedia.org/wiki/Fan_Hui 25

AlphaGo versus Fan Hui

AlphaGo won 5:0 in a formal match in October 2015.

[AlphaGo] is very strong and stable, it seems

like a wall. ... I know AlphaGo is a computer,

but if no one told me, maybe I would think

the player was a little strange, but a very

strong player, a real person.

Fan Hui

Fan Hui 26

Lee Sedol “The Strong Stone”

� the 2nd in international titles

� the 5th youngest (12 years 4 months) to become

a professional Go player in South Korean history

� Lee Sedol would win 97 out of 100 games against Fan Hui.

� biological neural network comparable to Fan Hui’s (in number

of neurons and connections)

https://en.wikipedia.org/wiki/Lee_Sedol 27

of neurons and connections)https://en.wikipedia.org/wiki/Lee_Sedol 27

I heard Google DeepMind’s AI is surprisingly

strong and getting stronger, but I am

confident that I can win, at least this time.

Lee Sedol

...even beating AlphaGo by 4:1 may allow

the Google DeepMind team to claim its de

facto victory and the defeat of him

[Lee Sedol], or even humankind.

interview in JTBC

Newsroom

Lee Sedol

interview in JTBC

Newsroom

Lee Sedol

interview in JTBC

Newsroom

AlphaGo versus Lee Sedol

In March 2016 AlphaGo won 4:1 against the legendary Lee Sedol.

AlphaGo won all but the 4th game; all games were won

by resignation.

The winner of the match was slated to win $1 million.

Since AlphaGo won, Google DeepMind stated that the prize will be

donated to charities, including UNICEF, and Go organisations.

Lee received $170,000 ($150,000 for participating in all the five

games, and an additional $20,000 for each game won).

https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol 28

by resignation.

Who’s next?

http://www.goratings.org/ (7th May 2016) 28

AlphaGo versus Ke Jie?

� the 1st in (unofficial) world ranking list

� the youngest player to win 3 major international tournaments

� head-to-head record against Lee Sedol 8:2

� biological neural network comparable to Fan Hui’s, and thus

by transitivity, also comparable to Lee Sedol’s

https://en.wikipedia.org/wiki/Ke_Jie 29

I believe I can beat it. Machines can be very

strong in many aspects but still have

loopholes in certain calculations.

Ke Jie

Now facing AlphaGo, I do not feel the same

strong instinct of victory when I play a

human player, but I still believe I have the

advantage against it. It’s 60 percent in

favor of me.

Ke Jie

Even though AlphaGo may have defeated

Lee Sedol, it won’t beat me.

Ke Jie

favor of me.

Ke Jie

favor of me.

Ke Jie

Conclusion

Difficulties of Go

� challenging decision-making

� intractable search space

� complex optimal solution

It appears infeasible to directly approximate using a policy or value function!

Difficulties of Go

AlphaGo: summary

� Monte Carlo tree search

� effective move selection and position evaluation

� through deep convolutional neural networks

� trained by novel combination of supervised and reinforcement

learning

� new search algorithm combining

� neural network evaluation

� Monte Carlo rollouts

� scalable implementation

� multi-threaded simulations on CPUs

� parallel GPU computations

� distributed version over multiple machines

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

AlphaGo: summary

learning

Novel approach

During the match against Fan Hui, AlphaGo evaluated thousands

of times fewer positions than Deep Blue against Kasparov.

It compensated this by:

� selecting those positions more intelligently (policy network)

� evaluating them more precisely (value network)

Deep Blue relied on a handcrafted evaluation function.

AlphaGo was trained directly and automatically from gameplay.

It used general-purpose learning.

This approach is not specific to the game of Go. The algorithm

can be used for much wider class of (so far seemingly)

intractable problems in AI!

Novel approach

Thank you

for the music

of your attention!

Happy to be here,

in your DISCO group!

Thank you

for the music

of your attention!

Happy to be here,

Thank you

for the music

of your attention!

Happy to be here,

Thank you

for the music

of your attention!

Happy to be here,

Thank you

for the music

of your attention!

Happy to be here,

Backup Slides

SL Policy Network: Accuracy vs. Win Rate

Small improvements in accuracy led to large improvements

in playing strength

Silver et al. 2016

Evaluation Accuracy in Various Stages of a Game

Move number is the number of moves that had been played in the given position.

Each position evaluated by:

� forward pass of the value network vθ

� 100 rollouts, played out using the corresponding policySilver et al. 2016

Input features for rollout and tree policy

Silver et al. 2016

Selection of Moves by the SL Policy Network

move probabilities taken directly from the SL policy network pσ (reported as a percentage if above 0.1%).

Silver et al. 2016

Selection of Moves by the Value Network

evaluation of all successors s′ of the root position s, using vθ(s)

Silver et al. 2016

Tree Evaluation from Value Network

action values Q(s, a) for each tree-edge (s, a) from root position s (averaged over value network evaluations only)

Silver et al. 2016

Tree Evaluation from Rollouts

action values Q(s, a), averaged over rollout evaluations only

Silver et al. 2016

Percentage of Simulations

percentage frequency with which actions were selected from the root during simulations

Silver et al. 2016

Results of a tournament between different Go programs

Silver et al. 2016

Results of a tournament between AlphaGo and distributed Al-

phaGo, testing scalability with hardware

Silver et al. 2016

AlphaGo versus Fan Hui: Game 1

Silver et al. 2016

AlphaGo versus Lee Sedol: Game 1

https://youtu.be/vFr3K2DORc8

https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

AlphaGo versus Lee Sedol: Game 2 (1/2)

https://youtu.be/l-GsfyVCBu0

https://youtu.be/qUAmTYHEyM8

https://youtu.be/yCALyQRN3hw

https://youtu.be/mzpW10DPHeQ

alphago for disco group of eth zurich

Science