Neural Network Implementation of Poker AI

Christopher Kramer

Outline of Information The Challenge

Application, problem to be solved, motivation

Why create a poker machine with ANNE?

The Flop The hypothesis

Can a Poker AI run using only an ANNE?

The Turn The approach and methodology

How is this machine created?

The River Experimentation

How should it be tested?

The Showdown Conclusions and future work

Did it work?

The Challenge The approach focused on creating an AI capable of playing poker at a

reasonable level of play using Neural Networks. The difficulty:

Incomplete knowledge

Multiple competing agents

Risk management

Opponent modeling

Deception

Unreliable information

Creating solutions to address these difficulties can be applied to other problems with similar domains.

NOISE!

The Fold (Hypothesis)

The strengths of computers: Quick computation

Large memory

Purely logical

Using a series of machine learning techniques including neural networks, it is predicted that a machine will be able to play at a moderate level while ignoring typical features of poker programs such as statistical analysis and pot odds.

The Turn (Approach)

Problem split into a number of stages

• Hardcoded• Action chosen based off of known values

of starting handsPre-Flop

• Neural Net • 5 card inputPost-Flop

• Neural Net• 6 card inputRiver

• Neural Net• Player hand analyzed and given as set of

input valuesShowdown

The Turn (continued)

• Hardcoded• Action chosen based off of known values

of starting handsPre-Flop

• Information gathered from percentage of games won with starting hand from poker databases

• All hands in top 30th percentile were analyzed for characteristics which were hardcoded into if statements

• All hands below threshold were folded. • Those with characteristics in top 5th percentile had a

higher random chance of being raised

The Turn (continued)

• Neural Net • 5 card inputPost-Flop

• Inputs the unique id of each card in hand and play• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .3 indicated an action to raise• Below -.3 fold• Else, Call

The Turn (continued)

The Turn (continued)

• Neural Net• 6 card inputRiver

• Inputs the unique id of each card in hand and play• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .3 indicated an action to raise• Below -.25 fold• Else, Call

The Turn (continued)

• Neural Net• Player hand analyzed and given as set of

input valuesShowdown

• Cards analyzed for poker hands such as Flush, Fold, etc. and high card of each returned.

• Input nodes for each poker hand. Value is the value of the highest card in hand.

• Hidden nodes utilizing sigmoid function• Output using sigmoid function between -1 and 1• Output values above .5 indicated an action to raise• Below -.2 fold• Else, Call

The Turn (opponent modeling)

Opponent modeling generalized to fit into previous Neural networks Opponents action modeling taken into account by Neural nets due to

opponents actions changing the weights of neural network to account for their actions.

2 Factors Player’s calls

For each raise by opponent, have a higher required confidence to raise again

Players treatment of poker hand values

As player actions change, the confidence values of the Neural nets should change to reflect the modeling of the player

The River (experimentation)

Experiment was developed to overcome short term failure due to bad luck 1000 dollars given to each player. $10 bet .

Engines initial weights were determined experimentally for each ANNE Weights were assigned randomly and run for 10000 generations against a set of random poker hands

and card tables. Winning gave an expected value of 1, loss -1, and draw 0.

Network was then checked against 1000 random validation hands. A degree of accuracy was required for each ANNE

Required accuracy lowered as amount of information known decreases.

Once Neural Network was fit level of confidence its results were accepted as the Engines default weights.

Weights continue training during the course of game to profile player. Tested against a number of human opponents based on skill level to measure success.

The River (experiment results)

Beginners AI won 82% of games against 50 opponents

Moderate (play at least 1 game a month) AI won 20% of games against 20 opponents

Advanced (play at least 1 game a week) AI won 1 game out of 10 against 2 opponents.

Random (never folded) AI won 63.2% of games out of 1000.

100% won against Random opponent who could also fold

The Showdown (Conclusion)

The complete reliance on generalized Neural Networks for a system with extreme noise is not a viable option for higher levels of play.

While it did not shown significant advantage over moderate and advanced opponents, it did well against beginning opponents who knew how to play poker but did not do so often.

So much noise…. While a degree of noise was expected, generalized poker domain’s noise was is in excess of

10-40% (approx.)

This led to extreme cases of overlearning regardless of an extremely low learning rate.

While this was partially handled using confidence, the amount of folds late in the game due to incorrect early predictions led to the machine still failing to moderate opponents.

Was still shown to do well against a random opponent and beginners.

The Showdown (future research)

There are numerous areas for improvement Current systems adjusts slower than humans Possible GA implementation instead of Neural Network to avoid direct

noise issues. Statistical analysis and pot odds would help as inputs for neural

network.

Questions?