Abstract
In a game of Texas Hold ‘Em poker, players are constantly forced to make complex decisions based on factors that are both known and unknown. This is a challenge that is ripe to be explored using artificial intelligence concepts. My project, entitled “FuzzBot,” is an automated poker-playing bot that uses these concepts, particularly artificial neural networks and fuzzy logic, to play a game of poker from within a software package called Poker Academy. This program allows developers to code their own bots in Java, using their provided Meerkat API interface to allow the bots to interact with the software. The neural networks will be used for the purpose of opponent modeling, which is the crucial art of predicting opposing players’ action throughout a game. The implementation of this will be similar to the University of Alberta’s PokiBot opponent modeling scheme. This network will be trained using data acquired from observing actual live-cash games on the Internet. The fuzzy logic inference system will be implemented for the purpose of deciding on which strategy to use for each betting round, and also which specific move to make when it is FuzzBot’s turn to act. Each strategy will be represented by an object derived from the NewStrategy abstract class, which utilize poker wisdom and playing tips derived from books written by professional poker champions.
Acknowledgements
I would like to thank Dr. Carl Bredlau for his valuable mentorship and advice in the technical and formal aspects of this project, Dr. Dorothy Deremer for her encouragement, knowledge, and for making me a graduate assistant which has been key for me be able to complete my academic duties, and to Dr. James Benham for his continued support and advice, and to the entire computer science department staff for giving a guy with no previous computer science experience a chance! Much thanks to my girlfriend for her love and support, as well as enduring my World Series of Poker television marathons. I would also like to thank my mom, dad, and entire family for without their support and faith in me, I don't think I would have made it this far. Thanks to the online poker community for their insightful posts on forums around the web. Thanks to the creators of Poker Pro 2006 for giving me a license to use their product for no charge, which was key in the gathering of input data for this project. Much thanks goes to the Edward Sansonov and the creators of the JOONE software package for making their wonderful code available for me to use. Finally, I would like to thank the University of Alberta poker research group for making such a wonderful program to develop and test my bot with, as well as for publishing a multitude of poker research papers and information on their website, which proved to be critical in designing this project.
1.1 Summary
My project is an exploration of artificial intelligence concepts as they apply to making uncertain decisions and adaptive behavior. The focus will be on neural networks and fuzzy logic, and the goal is to create an automated poker playing bot designed for use with Poker Academy Pro, an advanced Texas Hold Em simulator that provides many features to players. One feature is the ability to create a poker bot in Java and place this bot at a table against other bots, as well as the player themselves. Most of the concepts covered in this project build upon previous work in the field by the University of Alberta's poker research group consisting of of students, professors, and professional developers. This group has produced and automated bot, called PokiBot, that has been incorporated into the commercial video game “Stacked With Daniel Negeranyu,” a game that is released for the PC, Xbox, and Playstation 2. PokiBot is very advanced and effective, but even the Alberta group has stated that there are areas that can be improved upon, and this project will explore these possibilities.,
The primary focus of my bot will be on two entities: the neural network that will be used for opponent modeling and predictions, and the fuzzy logic inference engine, which will be used to assess real-time game information and make strategy decisions based on the output. At the heart of my bot is a Java object called PokerBrain. This object serves as the nerve center for all the interacting data structures. The PokerBrain object takes in information dealing with decision factors in a game of Texas Hold Em, such as position and pot odds, and relays them to a trained neural network. The neural network will return a "probability triple," which is an array of doubles that represent the likelihood of what a particular player will do - either fold, check, or raise. This network will be trained using data collected from thousands of actual cash hands played over the Internet on such sites as PartyPoker and Paradise Poker. This data will be collected using a program called Poker Pro 2006, which is a commercial software package used to record a player’s and opponent’s move, The data is stored in an Access database. I have written a script that will parse this information, and prepare a training set of data to train the neural network.
Once the PokerBrain object has queried the networks and has a valid prediction of the opponents projected moves, this data will be used to query a fuzzy logic engine, which will in turn select a Strategy object for use in that particular betting round. Each Strategy object holds a series of Boolean conditions that are used to select the bot's next precise move. For example, if the fuzzy engine selects the "Slowplay/Checkraise" strategy, the player will first check, and then (hopefully) when the opponent bets, the Strategy object will then return a "Raise" constant, since this is the usual mode of a slowplay strategy.
Observing and evaluating performance should be a relatively straightforward task. Another feature of Poker Academy Pro is the ability to keep detailed stats on not only the human player's performance, but also of any bots brought into the game. The primary statistic of interest is small-bets-per-hand, which is a measure of the overall profitability of the bot. This measures how much money a player or bot makes for each hand they play in terms of the small blind amount, which is the required amount that one player at the table must put into the pot at the start of a hand in addition to the big blind. This figure can be considered a crude average money earned, and is the measuring stick for all of the Alberta group's bots and all other papers
1.2 Why Poker?
Despite his important role in the development of modern computing, Von Neumann was not a good poker player. He realized early on that playing this seemingly simple game according to probabilities alone is a strategy that is doomed to fail. Poker, he surmised, is not a game of numbers, but a game of clever observation, deductive logic, and pattern recognition, with a little bit of luck thrown in. So when he wrote his first paper on game theory[17], which later became a subject of study in economics and politics in classrooms around the country, the result was a whole new aspect of artificial intelligence, with the side effect of augmented skill at his favorite game.
Poker, he concluded, is not a game of mere numbers, nor is it a game of complete information. Games such as chess and backgammon are indeed games with complete information – that is, nothing is hidden from the players, and all possible moves can be calculated at any point in the game. In Texas Hold 'Em poker, it is impossible, and contradictory to the nature of the game, to have all information regarding each hand. Players play the game with their two dealt cards face down, and are not even necessarily required to show them at the end of the hand. Not only that, but a hand may end at any given time if enough players fold their hands, leaving almost all relevant information behind. Certainly, playing a game such as this according to a static and rigid statistical system is not wise.
Strategy for a game such as this is ripe and ready to be explored using the principles of artificial intelligence. If poker was the inspiration for Von Neumann’s invention of game theory, then it is even more imperative that the power of today’s computing machines be used to expand the strategic horizons of this popular and challenging game. Much of the technique involved in playing Texas Hold 'Em Poker successfully (that is, walking away with more money that you started with!) involves making decisions that are seldom easy and clear cut, since they are based on factors that are unknown to the decision maker. This represents the human element, and is something that is neglected in a rigid rule-based system. However, to a certain extent, some speculation based on probability can be made, since some information is available to the player that can be used to narrow down the possible scenarios. In this respect, some amount of calculation is definitely appropriate when making poker decisions. Therefore, a mixture of both schools of thought is paramount to success, and it is this mixture that suggests extended use of artificial intelligence techniques for poker strategy.
Participating in the process of combining artificial intelligence systems and Texas Hold Em Poker strategy is a strong way to become more proficient in both subjects. The neural network that predicts poker players’ moves during games created by the University of Alberta research group [6] has a staggering nineteen input neurons, with only three output neurons. Surely, building such a network must have been challenging, and it can only be assumed that the developers gained valuable experience. Not only did they learn much about the nature and structure of neural networks, but the results of the network itself revealed valuable knowledge about the average player’s behavior. Out of all the input neurons, only a handful of them showed to have any significant effect on correct prediction of a players move. In other words, some factors that may have been considered to be important when making decisions turned out to be insignificant, and a poker player can without a doubt make note of this and adjust their opponent prediction techniques when playing a real-life game.
Be this as it may, the main focus of this project is the computational aspect of my system. To be honest, I am a casual, not skilled, poker player. My playing experience is limited to a few late nights at my friend’s house for a friendly game with a twenty-dollar buy-in, which I have never finished in the money, much to my chagrin. As far as playing online, I never owned an account on any online site until I gathered the training data for the neural nets. I do occasionally enjoy watching televised poker games, and I have an enormous amount of respect for the professional players who always seem to do well in every tournament they play in (which goes to show that it is indeed a game of skill!). My approach learned a is therefore non-biased; I do not have any preconceptions on what I feel is the best way to play the game. All of the poker strategy involved in designing the fuzzy rule set is directly derived from books written by experts - men and women who have become very wealthy from playing well consistently. This system would be way too complex for me to use in a real-life game, and although I have indeed great deal about the game of poker in the process of completing this project, it would be extremely difficult for even the most astute poker player to properly absorb the constant flow of information needed to execute this system effectively in a real-world scenario. Thus, the main goal of the project is to to develop and apply aritificial intelligence techniques to poker simulations, to write an extensive and complex piece of code, and, finally, to add to the extensive online knowledge base involving the combination of poker theory and computers.
1.3 Previous Work
The main challenge in designing an automated poker player is the issue of opponent modeling. Opponent modeling is the art of determining what other players will do when their turn comes. As simple as it sounds, the task is quite difficult. Poker is a game of imperfect information and great chance. Unlike games like chess, the best move at any given time can be accurately calculated. Also, chess and other board games require the prediction of only one static opponent. In a game of Texas Hold 'Em, there may be as many as nine other players, and those players may get up from the table to be replaced by new players with totally different playing styles.
Thus, the bulk of the previous research [1] conducted by other institutions has focused on opponent modeling: if one can accurately estimate their opponents actions, all other decisions will fall into place. By far, the most prominent research group is the University of Alberta's Poker Research group. Their poker bot, Poki [7], has become the standard for all other bots due to its highly effective modeling system. At the heart of this system is a neural network that takes in nineteen different inputs and output three double numbers. These three numbers represent the network's prediction that the player will either fold, check, or bet/raise. This is named the "probability triple." The output is then used in conjunction with a weight table that represents the frequencies of players’ actions, and a prediction is made. The network is a multi-layer perceptron with four neurons in the hidden layer. The network is trained with data that the team has collected from observing hundreds of poker games on the IRC network. Later, the research group discovered that only a small set of the inputs actually had any significant effect on the output, and the network was later streamlined, yielding more accurate results.
Other techniques have been implemented by the Alberta team, and others, with varying degrees of success. One of which is the use of decision trees. The Alberta group used Paul Utgoff's Incremental Decision Tree Induction [12] software as a means of opponent modeling, yielding a probability triple similar to the familiar neural net. Although this was a bit easier to understand than the neural net, the Alberta group claims that the decision tree yielded slightly less accurate results. Another approach relies on game theory and bluff strategy, which has proven to be very effective. BluffBot is a system that tries to bluff, or "bully" its opponents into folding as much as it seems fit. This echoes many professional poker player's strategy, such as renowned author David Slansky, who feels that a player should bluff as much as possible [16]. This bot was entered into this past summer's Poker Bot Tournament held in Cambridge, Massachusetts, and emerged victorious over all opponents, including the Alberta's Poki bot.
In a paper written by two students at Indiana University[12], emphasis is placed of slightly different factors. These two individuals attempt to predict an opponent’s hand strength, and the variance of that strength.. When making an attempt to predict opponents, they used a case-based reasoning framework as a basis . Their observation was that using a player's recent conduct is not an adequate foundation for modeling, since players change their style often, sometimes from betting round to betting round. They also concluded that "a player's playing style does not deviate from long term averages by much, and tend to return to it." Instead of using some sort of global prediction case, they found it was more effective to base their knowledge set on styles of play, ranging from loose and aggressive, to tight and passive. This, instead of basing their knowledge on recent individual history, proves to be more accurate in opponent prediction.
While all these systems have shown to be worthy, there are some areas that perhaps could be improved. First, I feel that the training set used to train the Pokibot neural network could be more authentic. The data used to train the network was collected from a non-monetary source. IRC does not require actual cash deposits, so all the players have nothing to lose. When players have actual money at stake, the actions and behaviors of the players tend to change, and become more predictable, since they are playing more "by the book." What I decided to do was collect data from actual cash games, some of which have very high stakes with blinds as high as $200, and some only deal with pennies and nickels. I used a program called Poker Pro 2006 to collect this data from online cash-games on such sites as PartyPoker.com and Paradise Poker. This software keeps detailed information about each and every move and the circumstantial game information that surrounds it. All information is stored in an Access relational database, spread out across many tables. Although the data for each player's move is quite cumbersome and sometimes utilizes arbitrary constants to represent moves and actions, the act of writing a script to connect to this database and parse the multitude of information was less complicated than expected. The script produces a text file that serves as the training set to the neural network.
Another area that could be investigated is the aspect of adaptation. Adaptation is commonly perceived to be necessary to keep up with opponents changing styles of play. While the previous systems discussed may judge accuracy as a function of predicted opponent action versus actual action on a move-by-move basis, I approach this aspect a bit differently. My approach judges the success of prediction models on a round-by-round basis. The criteria for success will be based on the desired outcome of the round as a whole, not on individual actions. For example, if my system were to choose "bluff" as its river strategy, the only action that matters is whether or not all my opponents fold their hands. If I predict that two of my opponents are to fold, and only one of them actually does, where the other one actually raises, which would be disastrous if my hand was of poor value, then a 50% accuracy is not relevant. The goal was to get all my opponents to fold, not just a certain amount of them that would seem acceptable.
1.4. Requirements
FuzzBot, which is the name I have chosen for my bot, must be able to play a game of Limit Texas Hold 'Em within the Poker Academy software package. When properly registered at Limit table, FuzzBot will be able to play and compete against both the human player and any other automated bots active in the game. FuzzBot must also have the critical capacity to not only make decisions, but also to be able to evaluate those decisions, and make adjustments accordingly. This will be done by adjusting the membership functions representing the "desirability" scores in each Strategy object.
The main technical concerns for requirements involve proper versions of Java and Poker Academy, the software which will incorporate the system into a poker match. It is recommended that version 1.47 or above of the Java Runtime Environment be installed on the system hosting the game. Also, since FuzzBot was developed using version 1.5 of the Meerkat API, then version 2.0 or above of Poker Academy is required. All the necessary class files, as well as configuration files, must be packaged into a JAR file and placed into the bot folder in the Poker Academy install directory. Given that FuzzBot is designed to run autonomously in a specific and specialized environment, the requirements for human interaction during FuzzBot's in-game operations are minimal, if any.
Since Poker Academy does not support any debugging or testing for custom bots, a pop-up must be created upon initialization of a game within Poker Academy. This window will provide detailed information for debugging purposes, as well as to monitor the changes and choices being made within FuzzBot itself. First, a textArea object will act as a console to display log information in lieu of System.out calls. Second, a separate area will display the current values representing certain membership functions. Since each Strategy object will constantly be shifting these values, it will be very useful to keep track of them during a match. Also, some basic statistics involving the frequency of each Strategy being selected overall and by betting round will be displayed. Finally, since opponent prediction is crucial to success, one part of the display console will be allocated to displaying what FuzzBot thinks each opponent will do in their next turn.
2.1 General Architecture
The FuzzBot system is divided into three main layers. The first layer acts as the interface between the FuzzBot classes and the Poker Academy software. This class, called FuzzBot.java, implements the Meerkat API and is responsible for returning an integer representing the move that FuzzBot will make when its turn comes. Below this is the Brain layer, called PokerBrain. This layer acts primarily as a controller of all the various bits of information that FuzzBot considers when making a decision, and also manages all the various Strategy classes. Finally, there is the service layer, which is comprised of the NetManager, FuzzyDecider, and GameData classes. These classes import the JOONE, FuzzyEngine, and Poker libraries respectively, and all of them directly interact with the Brain layer.
Every time FuzzBot is called on to make a move during a game, the action() method of the FuzzBot class is called by Poker Academy. This method returns a field constant which represents the move. Since this is a game of limit Texas Hold Em, the only valid moves that can be returned are check, call, bet, raise, or fold. There are also constant values for posting blinds, but this is one automatically returned and not accomplished through the action() function call. When action() is called, and if FuzzBot has not acted yet for the round, the getStrategy() method of PokerBrain is invoked. It is at this point when all of the intelligence of FuzzBot gets activated. In pre-flop play, the process of deciding upon which Strategy to use for the round is determined by only a few factors, where the process of Strategy determination for all subsequent rounds varies from strategy to strategy. Each Strategy contains unique rules, inputs, and fuzzy variables, and when the FuzzyDecider class is called to select a Strategy, it uses these field values to query the FuzzyEngine to produce a "desirability score." This model is similar to Mat Buckland's [5] weapons selection system for bots in first-person shooting games, where each weapon returns a score based on remaining ammunition and the distance to the enemy player. Once each score has been returned to the PokerBrain object, it then selects the Strategy with the highest score that is above .500 and keeps that strategy for the duration of the beting round.
The Strategy class is abstract, and is used to derive all the various situational playing strategies that it can select upon first acting in a round. Each Strategy also has an ID number, which is used when the Brain layer selects a strategy module to be used. All Strategy modules also have a getMove() method, which is relatively straightforward. Some of these methods will always return the same value, such as the Fold() class. At this time, the implemented Strategy modules are Slowplay, Bluff, SemiBluff, Freeride, ValueBet, CheckRaise, and Fold.
Once a Strategy has been selected for the round, it is that Strategy object that becomes responsible for making the actual move for FuzzBot's turn. Each strategy has a different process for making this decision. Some are very simple, such as the Fold and Slowplay modules, while others such as Bluff and SemiBluff are a bit more complex. The general idea is that all good poker players base their actions on short-term, strategy-based reason, and not on a move-to-move basis. Therefore, it is central to the philosophy of FuzzBot's functionality that the process for deciding which strategy to use should be emphasized over the process of determining each single move.
Each Strategy module holds a number of linguistic variables that receive their input values from other components outside of the fuzzy domain. The most important ones involve getting opponent prediction scores using the neural net embedded in the NetManager class. This is done through the function calls getChanceOfCall(), getChanceOfBet(), and getChanceOfFold() that belong to the PokerBrain class. For example, the Bluff strategy needs to know how likely it is that the remaining opponents would fold if FuzzBot were to raise, along with other factors. So, it calls getChanceOfFold(), which returns a double value, which in turn gets fed into the Fuzzy Engine as the input to the chanceOfFold linguistic variable. All of these methods call on the NetManager to query the neural network to produce the "probability triple" for each remaining player in the hand. This is the same concept used in the University of Alberta's opponent prediction system. These values are then recorded for each player by passing the three values into the GameData object owned by the PokerBrain class.
The NetManager class is only responsible for returning an array of double values to the PokerBrain class. The PokerBrain uses this output from the neural net to calculate the chance of each remaining player making the move in question. Only the highest is returned to the Strategy class. This is because if one person is likely to perform the action being predicted, then this is all that needs to be known, since the chance of a strategy being activated rests on a unanimous decision by remaining players. With the Bluff strategy, for example, all players must be expected to fold their hands in order for the strategy to be a success. If even one player stays in the hand, the strategy fails since the goal is to drive all players out of the pot. So, the player who is most likely NOT to fold represents the value that is ultimately returned to the calling Strategy module.
Any other information that FuzzBot uses as input into the Fuzzy Engines is for the most part covered by the GameData class. GameData is in essence a container class that holds general game information that is updated with each move made by a player at the table. Since the player data and public game data are stored in various classes such as the GameInfo, PlayerInfo, and Hand classes, the GameData class pulls all the disjoint information together in one class, and also provides valuable methods for retrieving information easily, without having to create addition instances of the native classes. This GameData class is visible by both the interface and the Brain layers of the system, since the information being passed to it comes from many different places. The GameData class is also being constantly referenced by the Fuzzy Engine instances to gather input data for making a decision.
2.2. The Interface Layer – The Meerkat API
The University of Alberta Poker Research Group has created an extensive API called Meerkat that contains the required interface class for developers to implement with their bots. Developers must implement the Player class in order to have a bot play in a game. This Player class also has methods that can be used to determine which hole cards the player is holding, and in turn the hand ranking. The Meerkat API provides not only an interface for custom bots such as FuzzBot to interact with the Poker Academy software, but also a number of classes that provide valuable services and information related to any aspect of the current poker game. The most valuable is the GameInfo class, which is used to hold any data about the current game that is open to all players, such as the amount in the pot, the current betting round, and the cards on the board. Another very useful class is the PlayerInfo class, which is quite convenient for holding information about other players at the table, such as whether or not a player has acted in the hand, or if the player has raised in the current hand. HandEvaluator is a class that is used to assess the hand strength of a bot's hand against the board. Many of its methods use two instances of the Card class, representing the bot's hole cards, and can return the value in several different formats, such as how many hands this combination would beat, or simply as a percentage of winning. Finally, there are some minor classes that only provide basic services, such as the Card and Hand classes. Although the Meerkat API provides a multitude of information and convenient services to the developer, there is a need for some sort of centralization of this sometimes disjoint information, which gives way to the GameData class that is discussed later.
In order for any bot to work correctly from within the Poker Academy software, developers must implement the Meerkat API interface, which is provided by the University of Alberta Poker Research Group. The Player class holds the necessary methods that the developer must override. The first and most essential method is the action() method. As stated above, this method is called every time FuzzBot's turn is up, and returns an integer field value that represents the move. Most of the other methods are not as essential, but still must be implemented as is the standard for Java interfaces. init() is called before the hole cards are dealt. and is used to reset certain values in the GameData object, such as the round and bets-to-call. gameStartEvent() is called immediately after init() and it is here where the integer representing FuzzBot's position is assigned. This value should not be confused with the "seat" value, which remains constant for as long as FuzzBot is in its current chair. Position is assigned relative to where the "button," or the last player to act in the betting rounds, is seated, and this button usually rotates clockwise one player after each hand. The gameStageEvent() is called at the beginning of each betting round. The showdownEvent() method is called in the event that two or more players stay in the hand all the way to the end and a "showdown" takes place. In poker, this is when the first player to act is required to show their hand. If any subsequent players have a superior hand, they are required to show theirs as well, but if they know they do not have a better hand, then they can "muck," or discard their hole cards without having to show them, which is usually what happens. The showdownEvent() is very useful for recording what opponents were holding during the hand, because developers can derive the Card objects that the showdown players reveal. Finally, there is the winEvent() method, which the called after a player has been deemed the winner of the hand, and is used to record which player and how much has been won. Sometimes there is a "split pot" where two or more players end up receiving chips, and this method will indicate this.
While the Player class is essential for developers to implement their bots, it is the GameInfo that proves to be the most useful out of all the Meerkat classes. GameInfo mostly is comprised of methods that return booleans, but the ones that return other data types are extremely useful. The methods that get used most often are the one that involve opponent's actions in the current round. Using the getBetsToCall() method, an integer is returned that represents the amount of money that FuzzBot must place into the pot to remain in the hand. This is needed to calculate pot odds and expected value, which is critical to formulating proper strategy, and is even more important in Limit games where players’ actions are known to be more aggressive. Although there is a method that calculates pot odds directly, this single value is used most often in querying the neural network for opponent modeling purposes, and has been shown to be a key determinant in predicting opponents’ actions. Another key method is the getPlayer() method, which returns a PlayerInfo object (discussed later). Any information that is specific to a certain player is gained through this method, and is used all throughout the FuzzBot decision-making process. Then there are the methods that indicate the nature of a player's actions in the betting round. The getUnacted() method indicates how many players have yet to make a move during the current beting round. When determining FuzzBot's position, the Fuzzy Engines use this method to determine the all-important factor of whether it is in early (among the first players to bet), middle, or late (player is, or is close to, the button). This is another factor that has proven to have a great influence on player's actions and is considered to be a deciding factor in strategy determination by all poker experts. In calculating FuzzBot's hand strength and value, it needs to know the board cards, or the cards that are face up on the table. This information is gathered through the getBoard() method, and returns a Hand object, which contains the relevant number of Card objects. Rounding out the array of methods are the ones that return booleans, such as isTurn() or isActive(), which return whether the current betting round is the turn and whether or not a player is still active in the hand, respectively.
2.3. The Brain Layer – PokerBrain
The interface layer, called FuzzBot.java, is responsible for implementing the required methods to interact with the Poker Academy software, but does not encapsulate all the key elements of the FuzzBot system. If all the various artificial intelligence entities were directly accessed in this class, it would become very cumbersome and difficult.. Therefore, all of the main functionality is managed in a class called PokerBrain. The PokerBrain class acts as a central nerve center for the Fuzzy Engines within the Strategy modules, GameData, and NetManager components, hence the name. The FuzzBot class creates a single instance of a PokerBrain object that holds all the methods needed to play a game. Whenever FuzzBot is called on to make a move in the interface layer, it makes a call to PokerBrain to first choose a strategy, and in turn, make a move. It is the getMove() method that returns an integer constant to Poker Academy, which is dependant on the result of the getStrategy() method that is called at the beginning of the round. Within the PokerBrain class are a multitude of methods that interact with the external components. The method queryNets() is used by the Strategy modules to get results from the trained neural net, and is used by the methods getChanceOfFold(), getChanceOfCall(), and getChanceOfBet(). Also, there are methods that interact with the GameData object to provide a straightforward and centralized way of getting information regarding opponents’ games styles, as well as the current read on the grip (looseness/tightness) and style (passivity/aggressiveness) of the game as a whole. The main reason for this is to pull together all of the disjoint bits of information from the Meerkat information classes as well as the custom artificial intelligence classes into one class to make the code as a whole more writable and readable.
The most important methods inside the PokerBrain object are the getMove() and getStrategy() function calls. The PokerBrain class first creates new instances of each and every Strategy module and places them into an ArrayList. When called upon to choose a strategy at the start of each betting round, PokerBrain will call each Strategy getDesirability() function, which returns a double value. PokerBrain then chooses the one that has the highest score that is above .500 and stores this index to be used when it is time to receive a move from that strategy. If no score above .500 is returned from the Strategies, then the default Strategy, Freeride, is chosen for the round. Freeride simply has FuzzBot stay in the round as long as it is free to do so – that is, stay in until an opponent bets, and if one does, then fold. Once an index value is chosen, its value is assigned to a static field value and is used when the getMove() function is called via the action() method in the interface layer. Then, the PokerBrain calls the proper entry in the Strategy ArrayList and calls its getMove() function for that Strategy, which returns another integer constant that is passed to the interface layer. Since all of the logic for move determination is encapsulated in the Strategy modules themselves, PokerBrain is only responsible for providing game and opponent information to the modules, and not responsible for the decision making process.
This design pattern is to emphasize the concept of the fuzzy decision making process, which is that a strategy is only chosen if it is indicated to be a "good" strategy, or one that can be applied to the current game circumstances. Note that the desirability scores are not calculated on what the optimal winning strategy is perceived to be at the given time. Instead, the rules for the fuzzy engines within the Strategy modules are designed to detect which Strategy best fits the current situation. Sometimes a Strategy will be chosen, only to then return a "fold" constant upon being asked for a move. This will be discussed in greater detail in the "Strategy" section later. It may not be appropriate for any type of strategy to be used in a given situation, so if no Strategy indicates this, then none will be chosen, and the default Strategy becomes activated.
Another essential service that the PokerBrain class provides is in the domain of opponent modeling. When the Fuzzy Engines within the Strategy modules need to know the likelihood of a certain action the remaining opponents actions may make, they make a call to PokerBrain getChanceOfFold/Bet/Check() method. First, PokerBrain calls its private method queryNets(), which takes in various bits of public information including how many bets there are to call, if an ace or a king is on the board, the last action taken by the player, and if a straight or flush draw is possible. All this information is passed to the NetManager class and serve as the inputs to the neural net. The net is queried, and returns the probability triple, representing the chance of that player either folding/ calling/checking/ or betting/raising. Each value is passed to the GameData object and gets recorded for each individual player if it is the highest or lowest recorded triple score for that player. Depending on which move PokerBrain is trying to predict, the appropriate output of the triple is then fed into a Fuzzy Engine as an input. The membership functions for this Engine are determined by the minimum and maximum triple score logged in the GameData class for that specific opponent. This allows FuzzBot to adjust to an opponent's changing style, and make predictions that are specific to that player’s style and grip. The other inputs to this Engine are the player's style and grip, which are also taken from the GameData object. Once again, the membership functions for these linguistic variables are derived from the GameData object, and are calculated using the overall game style and grip. So, the inputs from the probability triple, player’s style, and player's grip are put into the Engine, and a double value is returned to PokerBrain. The PokerBrain object repeats this process for each active player, and the highest score is then passed to the Strategy as an input to its Fuzzy Engine.
2.4 The Domain Layer
The FuzzyDecider class bears the responsibility of deciding which NewStrategy object will be selected as the strategy to be used by FuzzBot for any particular betting round. It imports the fuzzy logic package designed by Professor Edward Sanzonov [15], and uses this to create a fuzzy logic inference engine, as well as all the necessary components required to efficiently run the engine. One instance of this class is owned by the PokerBrain object that resides in the "brain" layer of the architecture. The main concept is for one class to hold all of the methods and attributes required to implement a fuzzy logic system.
FuzzyDecider owns some very important attributes that are critical to making a viable strategic decision. The first is the preFlopPairTypes String array. This is a text representation of all possible hole card combinations, ranked from best to worst. At the top of the list are pocket aces, ace-king, and other highly desirable hole cards, while the hole cards with the lowest chance of winning are at the end. Any poker book that I have studied for this project has very strict rules as to which hands to play on the flop, and which ones to fold. This attribute makes the decision process for pre-flop strategy very simple. By using this attribute, the FuzzyDecider class can assign an integer value representing hole card rank, which can easily be processed without the need for any Meerkat API function call.
Two other essential attributes that FuzzyDecider contains are the FuzzyBlockOfRules and FuzzyEngine instances. These are critical components for the fuzzy package to work, since ultimately it is the FuzzyEngine class that performs all the fuzzy operations, and uses the FuzzyBlockOfRules class to parse the rules and check for any errors regarding the wording of the rules themselves (misspellings of variables, NoRulesFiredExceptions, etc). Of course, these two components alone are useless without the rules themselves, as well as the linguistic variables and their inputs. These are supplied by the NewStrategy modules that are passed by the ArrayList from PokerBrain.
When it is time for FuzzBot to select a strategy at the start of every post-flop betting round, the PokerBrain instance calls the getStrategy() method of the FuzzyDecider class, by passing the ArrayList attribute that holds all the NewStrategy modules, as well as a reference to itself, and FuzzBot's two hole cards. Then, FuzzyDecider determines if the current betting round is pre- or post-flop. For now, let us assume that the current betting round is post-flop. Now, the getPostFlopStrategy() method is called. The FuzzyDecider then picks the first strategy module in the ArrayList. Then, this module is passed to the getDesireScores() method, which is where all of the fuzzy evaluation is performed. A new instance of a FuzzyEngine is created, and then the getInputs() method of the current strategy module is called to update the input values within the modules so that it is using the most current information regarding the game. Then, a loop is called that registers each module's fuzzy linguistic variable to the engine. The FuzzyDecider class then calls the getRules() method of the strategy module, which returns a String array representing that modules unique fuzzy rule set. This block of rules then gets registered via the FuzzyBlockOfRules attribute, and then gets parsed for any errors. Then, the input values get set for each input linguistic variable, and the engine gets put into action by calling the evaluateBlockText() method or the evaluateBlock() method. The only difference between the two is that the former returns a string that displays all the rules fired and their scores, which is extremely helpful during the debugging and testing process. Finally, the output linguistic variable is acquired by the getOutputLV() method, and gets defuzzified. This value becomes the "desirability score" for the current strategy, and is recorded by the FuzzyDecider. This process repeats for each strategy in the list, and the strategy with the highest score becomes the strategy for the round. Ultimately, an integer value that represents the strategy's position in the ArrayList is returned to PokerBrain, which then uses this to select the active strategy from the list.
For pre-flop strategy selection, the process is much simpler. While the University of Alberta group claims that preflop play is not critical to proper play, all of the books I have read on limit Hold 'Em stress strict rules to decide whether or not to play a hand. As a compromise, I have decided to implement a simple system to make this decision based on two factors: hand rank and position. The closer that FuzzBot is to the button, the more likely it is to play a hand. Once position has been determined to be either early, middle, or late, the ranking of the hole cards is determined using the getPreflopHandRank() method. This method gets the string representation of the hand by calling the getCardString() method of the Hand class, passing in the two Card objects that FuzzBot holds for the current hand. Then, FuzzBot searches for a match of this string against all the string values in the preFlopPairTypes array, and then returns its position. The lower the integer, the higher ranked the hand is, and therefore, the more likely that FuzzBot will play the hand. How exactly it plays the hand, that is, whether or not it limps in (checks), calls or places a bet, or raises depends on the precise hand rank and the position.
The NetManager class is probably the most complex out of all the components in the FuzzBot system, since not only is it elaborate in its precise implementation, but also in its creation and training. This class contains all the packages and methods needed to implement a neural network using the JOONE visual neural network editor [13]. The NetManager class is responsible for all aspects of opponent modeling and move prediction. JOONE provides a framework for creating, testing, and training all sorts of neural networks, and in the case of FuzzBot, is used to make the core of the neural system. FuzzBot uses a feed-forward, back propagation artificial neural network with eight input neurons, three hidden-layer neurons, and three output neurons. The sigmoid activation function is what is used for the neurons since only positive values will be processed. By using the JOONE GUI editor, the creation of a neural network such as this becomes much more streamlined than coding one completely in Java due to the ability to serialize a trained network. The file trainedNet.ser may be small and unassuming, but this tiny file represents a multitude of poker knowledge and real-world habits and tendencies of actual poker players playing for cash on the Internet.
While designing the architecture of my network, I considered the previous efforts of programmers who have done the same before me. The University of Alberta Poker Research Group has trained its neural network, which is the central part of its PokiBot opponent modeling system, by collecting data from games played on the IRC channel #poker [6]. The group observed thousands of games played by real players, and recorded as much data as it could regarding any public game information relevant to each individual player at any given time. Their network uses a staggering nineteen input neurons, each representing a unique decision factor, such as pot odds, previous move, and whether or not an ace is on the board. However, the group discovered that only a small amount of those input neurons had any significant effect on the output. Given this, I chose to use only these inputs, which totals to be eight, while using the same amount of neurons for the hidden layer. These inputs are the numbers of players left in the pot, the number of bets to call, whether or not an ace or a king is on the board, how many players have not yet acted in the round, whether the last move made was a bet or a raise, and how many bets did the previous player to act have to call.
The output of my net is very similar in structure to the Alberta's group, for it produces a "probability triple." Each of the three double values produced by the network when queried represents a rough estimation of the probability of a players move: fold, check/call, or bet/raise. Since the output of any neural network may produce esoteric values, the Alberta Group uses these values and compares them to the actual frequency of opponents’ moves in the game, and then computes a final probability. For example, the network yields a [0.0, 0.7, 0.3] score for Player A, while Player A has shown to fold, call, and bet 15%, 45%, and 40% of the time respectively, The triple then acts as a weight table and modifies the latter scores for the player.
I chose to process the output of my neural net a bit differently, and in a very experimental way. FuzzBot records the highest and lowest scores for each output neuron throughout the game. It then uses these values to define the membership functions for the foldScore, checkScore, and betScore linguistic variables which are held by various NewStategy modules. Then, the actual scores from the triple are used as inputs for these variables. Upon defuzzification of these variables, a double value is produces which acts as the rough probability of an opponent's predicted move. For example, the Bluff strategy module wants to know the chances that all remaining players will call a bet that FuzzBot may make. When it is time for the FuzzyDecider to evaluate the desire score for this module, it will call the getInputs() method of the Bluff module. This in turn calls the getChanceOfCall() method through the PokerBrain class, and the NetManager class springs into action. The NetManager class gathers the input data for the neural network and queries it once for each remaining player in the hand, and an output triple is produced. For this case, the NetManager is only concerned with the second value of the triple that represents the chance of call. The highest gets returned to the FuzzyDecider class, and is used for the input for the second linguistic variable. This linguistic variable then recomputes its membership functions via the getLVs() method using the highest and lowest overall scores recorded from the triple during the whole game. The fuzzy engine then gets queried using the latest minimum/maximum probability scores for the membership function regarding the chance of someone calling a bet, and the output from the neural net as the input. The process is identical for the getChanceOfFold() and getChanceOfBet() methods, and at the time of this writing, still remains a very experimental method.
Probably the most difficult phase of implementing this network was the process of training. As mentioned above, the University of Alberta group chose to use only play data for its input set – that is, games played not for actual cash, but for points on IRC. While they claim that despite the lack of financial risk, the players still act as if it were an actual live game, I chose to gather the inputs for my training set from data involving actual cash play in internet poker games. Sites such as PartyPoker.net and ParadisePoker.net are great sites to observe player in all sorts of games, where the blinds range from pennies and nickels, to hundreds of dollars. Also, there is a very handy program called Poker Pro 2006 that runs in the background when you play at these sites that gathers data about opponents play move-by-move, as well as your own. This program turned out to be a wonderful tool in gathering the data, but did take quite a bit of time to figure out how to use it effectively. PokerPro 2006 records each and every move in a game for each player in an Access database file. The problem is that there are over 14 different tables in the database, and most of the data is recorded using integer codes. For example, the number one represents a fold, two represents a call, and so on. All the information that can be used for training the nets are scattered among the various tables. Any information not entered into the database directly, such as previous move and pot odds, had to be processed manually. To top it off, all data must be properly formatted to the specifications of the JOONE framework, which means all inputs must be double values, separated by semi-colons, with the validation data placed at the end of the string. A typical input field looks something like this:
0.0;0.2;0.2;1.0;0.6;1.0;0.5;0.5;0.0;1.0;0.0
Therefore, it was necessary to write a script that would parse and process the wealth of disjoint data, and turn it into a series of records similar to the one shown above. The file createInput2.java accomplishes this, and is displayed in appendix C.
The result is a text file that represents nearly 4,500 moves made by players in live Internet cash games. Once this file has been successfully created, the training of the neural net is a simple affair. Using the JOONE GUI editor, I can select the text file through a pop-up window. The file will work properly, as long as the amount of input values match the number of input neurons, and likewise with the output values and number of output neurons. Then, I attach a learning synapse to the output layer, set the number of epochs, and begin the training process all from within JOONE.
Once the network has been trained, it needs to be integrated into the FuzzBot system. JOONE can export the neural net as a serialized file. Then, this file is brought into the system through java.io and joone.io package functions. In order to query the network, all input synapses are removed through the removeAllInputs() method. They are replaced by a MemoryInputSynapse object, which accepts input values through the text file produced by the script. The same is done for the output synapses. Then a Monitor object is acquired by the getMonitor() method, which controls all the querying activities. The getNextPattern() method of the MemoryOutputSynapse is called, which returns an array of three double values that represents the probability triple, and this is what gets returned to the calling method.
2.4 The NewStrategy Abstract Class
The core of the FuzzBot system lies in its strategy modules. Each time FuzzBot is called to make a move for the first time in a round, the PokerBrain object calls the getStrategy() method, which activates the FuzzyDecider. The Fuzzy Decider hosts all the required classes to implement a fuzzy logic inference engine. When the Fuzzy Decider becomes activated, it cycles through each strategy module’s rules, inputs, and linguistic variable set, and uses them to run an instance of a FuzzyEngine class. Each strategy has its own rules and linguistic variables, and are initialized in their constructors. As of now, the strategy modules being used by FuzzBot are called Slowplay, Bluff, CheckRaise, FreeRide, and ValueBet. Each module’s field attributes are designed around the teachings of poker strategy books, written by experts such as Doyle Brunson and David Slansky[16]. Therefore, the field attributes for each unique Strategy module are crucial to the success of FuzzBot’s playing style.
The sequence of events regarding the choice of strategy is designed to be quick, expandable, and memory efficient. All strategies are contained in an ArrayList object within PokerBrain, created upon initialization. This list is passed from PokerBrain to the FuzzyDecider class when getStrategy() is called. Then, Fuzzy Decider takes the first strategy’s rules, linguistic variables, and inputs, and uses them to create an instance of a FuzzyEngine, and then registers and parses the rule array. Once this has been completed, the FuzzyDecider then calls the current strategy’s getInputs() method, which uses PokerBrain as a parameter to update any of the information that will be used as the input values. Once FuzzyDecider has these values, it then uses them to query the engine, and then returns the defuzzified value which represents the strategy’s desirability score. FuzzyDecider repeats this process for each strategy in the list, and records the id number of the strategy with the highest score to PokerBrain. When PokerBrain now has a strategy to use for the round, all it has to do is call the strategy’s getMove() method, which returns an integer constant which represents the action to be taken by FuzzBot in the current game.
The NewStrategy abstract class provides a framework for implementing real-world limit Hold-‘Em strategies into the FuzzBot system. The most important aspect of the strategy modules is the String array that holds the rules to be used in the fuzzification process. These rules ultimately dictate whether or not a strategy fits the current situation in the game, and influence whether or not that strategy will be chosen for the current gaming round. They are carefully designed to not necessarily indicate what the best strategy is at the moment, but which strategy’s philosophy is applicable to the game. Once a strategy is chosen by the FuzzyDecider, FuzzBot will stick to it for at least the entire round. This is due to the fact that many poker strategies prepare a player for multiple actions in a round, such as CheckRaise, which simply instructs FuzzBot to first stay in the pot with a minimal bet, with the intention of raising any player that ups the bet later in the round. A move-by-move strategy system does not make sense when reading expert books and articles, and most of these books divide up their strategy sections into which betting round the play is considering.
In addition to the fuzzy rule set, there are other essential components to the NewStrategy class. Each Strategy has three or more linguistic variables associated with it, as well as their corresponding membership function parameters. At the minimum, each module must implement two linguistic variables for the inputs, and one for the result. A typical linguistic set would be one for hand strength, one for FuzzBot’s position, and one to represent the “desirability score.” It is this variable that gets defuzzified, and becomes that strategy’s score, which gets compared to the other scores by the Fuzzy Decider. In turn, the design of the membership function for the output variable is very important, so that the strategy does not produce an inaccurate score, which would lead to it being chosen either too often, or not enough.
Another important aspect is the getInputs() method, which is called when it is time for the FuzzyDecider to query the fuzzy engine and assign input values to the fuzzy linguistic variables. Since the information influencing FuzzBot decisions are constantly changing with each and every move its opponents make, it is crucial to make sure that the proper values are being used to query the engine. This method updates this information indirectly via the PokerBrain object, which in its implementation is getting all relevant information from an instance of the GameData class. Also, there is a method that returns a Boolean value called useFuzzyEval(). At the time of this writing, there are some strategies that will not use the fuzzy engine to produce a desire score. One of these is the ValueBet, which basically states that if the expected value for FuzzBots hand is positive, and there are fewer than three opponents left, then FuzzBot should at least check. Strategies like this one will return “false,” while the other return true. Should the FuzzyDecider begin to evaluate one of these strategies, it will know to bypass the fuzzy decision process, and use the strategy’s getDesirabilityScore() instead. Finally, there is the deceptively simple getMove() method, which is ultimate what is called upon when FuzzBot needs to make a move in an actual game. It seems simple, and some of the strategies do have simple methods for this that only return one value every time, but the more complex versions, such as Bluff, are difficult to design and often debatable as to their accuracy and effectiveness. For Bluff, the goal is to get FuzzBot’s opponents to fold, but if they don’t and choose to stay in the pot (or even worse, raise!), then FuzzBot needs to decide whether to fold or stay in, and this can be a very difficult decision to make, since many factors about the nature of the current game and its players are taken into consideration.
Each module is modeled after a standard playing strategy used by professional poker players in real-life games. While each strategy’s exact details and philosophy may slightly differ depending on whose book you are reading, the situation where one would use them and their desired outcomes are generally agreed upon. Without turning this part of the paper into hash of the poker book section of a bookstore, I will attempt to briefly describe each strategy.
Slowplay: In Slowplay, the situation is where FuzzBot has a strong hand or one that is likely to improve, and wants to drive as much money into the pot as it can without scaring players into folding by betting large or raising. Generally, the Slowplay strategy will cause FuzzBot to check when it can, and if raised, will make a decision based on hand strength and potential.
Bluff: Bluffing is, in essence, lying. FuzzBot will have a hand that isn’t very strong and is unlikely to win. However, if it is the Turn or River stage, and only a few weak and passive players remain, FuzzBot will bet or raise to act as if it has acquired a strong hand. The intention is to get the remaining players to fold, and the strategy is ineffective if any players remain in the hand, since most often players who bluff have the lowest ranked hands, if any. As stated above, although the initial choice of action when FuzzBot decides to bluff is easy, the choice of what to do should an opponent call or raise. As of now, FuzzBot looks at its pot odds and how many players are left in the pot when deciding what to do. When faced with a bet after bluffing, if the pot odds are in its favor, and there are no more than two opponents left, then FuzzBot will call, otherwise it will fold.
SemiBluff : If bluffing is acting as if your hand is strong when it cannot possibly win, a semi-bluff is the same, but your hand does has the potential to become better. A card that may show on the board during the next round that will increase the value of your hand is called an “out.” Using the “ppot” algorithm that is provided by the Meerkat SDK, FuzzBot can determine the “potential” of its hand, and using the “grip” (whether a game is “tight” or “loose”), will determine whether a semi-bluff is in order.
CheckRaise: Also called “sandbagging,” a check-raise is an attempt to lure players into calling bets that do not have correct pot odds. The first move is to check, and hopefully another player will bet. Then, for the second move in the round, the idea is to then raise the bettor in hopes that all remaining opponents will call, thus driving more money into the pot, and forcing opponents to make bad decisions. Most check-raises happen after the flop. The key to success of this strategy is to have a strong hand, and also be sure that an opponent who acts after FuzzBot will bet. If not, then FuzzBot will in turn give a “free card” to its opponents – that is, will have let opponents who are likely to lose to FuzzBot remain in the hand without having to bet.
FreeRide: This module is not really a strategy, since all it says is for FuzzBot to remain in the hand as long as it can for free. Once faced with a bet, FuzzBot will always fold. Typically, FreeRide becomes the active strategy when all other strategies have returned a low desirability score.
ValueBet: Pot odds are an important factor in Limit Hold Em. Generally, if the pot odds are good, and in turn the expected value is favorable, this usually implies that the player should remain in the pot, perhaps even raise, even if they don’t have a very strong hand. The ValueBet module checks the expected value and how many players are left, and if the situation is favorable, will lead FuzzBot to remain in the game.
3.1 Results
In Poker Academy, it is very easy to test a custom bot. You simply set the bankroll of the human player to zero, and set the game "Auto Deal" option to automatically deal a new hand after each one ends. Then, you simply sit back and watch your bot in action. Poker Academy keeps detailed stats about your bot, but the ones of most important are small blinds won per hand, and the distribution of actions taken. the former is a measure of how much money FuzzBot wins per hand as a function of the size of the small blinds for that match. The latter is a breakdown of how many times FuzzBot folds, checks, calls, bets, and raises in terms of percentages. For small blinds per hand, FuzzBot rate is about -.25. While this obviously implies it is losing more than it is winning, it could be much worse, and considering the fact that it is playing against professionally-designed bots that are made to mimic champion players like Doyle Brunson and David Slansky, it definitely shows potential.
After about one thousand hands simulated, FuzzBot seems to be folding about 20% of the time. When compared to the other bots, this rate appears to be low. This could be the key to increasing the winnings in the long-term, since Limit Hold 'Em strategy heavily emphasizes careful selection of hands to play into the flop. Also, FuzzBot appears to be raising much more than the other bots at the table. Combine this with the fold statistic, and it seems like FuzzBot is being too aggressive.
In the game of poker, you win some, you lose some. When it comes to describing FuzzBot's success at the tables, this is pretty much the case. While FuzzBot hasn't been the last bot standing at any of the simulated tables in Poker Academy, it hasn't been the first one to go as well. Most of the simulations have been conducted with a full ten players sitting in, and some with only four or five opponents. FuzzBot appears to do better with fewer players, implying that its over-aggressive style could be better suited for smaller size tables. Certainly, it is much more difficult to bluff successfully with nine other people to fool, so perhaps this could account for the increased success at the smaller tables as well.
3.2 Conclusions
Aside from the fact that modeling poker opponents is a very complex affair, it can be concluded that there is great potential for the use of fuzzy logic in poker decision making. Without a doubt, the process of making decisions in a poker game is very much indeed "fuzzy," that is, complicated by the mass amount of known and unknown data without any clear correct decision. For a fuzzy logic engine to compete against other kinds of bots and human players, the key is in the design of the rules. If the rules are not built around a solid knowledge-base, then the whole concept can be rendered useless.
Another critical aspect is the ability for a computer-controlled poker player to adjust to the games style and opponents. This has proven to be the most difficult part of designing FuzzBot. One thing that is almost impossible to gauge in a computer game of poker are tells. Tells are usually indicated by opponents body language and mannerisms, and these are all lost when playing computer opponents. In this attempt to design a poker player that emphasizes human patterns of thought in lieu of rigid statistical and probability laws, the inability to incorporate and quantify physical opponent tells into the fuzzy system could be FuzzBot's main disadvantage. Tells can give away valuable information about what an opponents is holding, and without tells, one can only make mathematical decisions. Since FuzzBot is not based on mathematical thinking, it can only speculate on opponent' habits based on trends and past behavior of those opponents. It certainly is risky, and at this point, FuzzBot takes on those risks with varying degrees of success.
I believe that the concept of integrating fuzzy logic and neural networks into a poker playing system can definitely work, but as the Albert research group has shown, the success of such a marriage must be brought about by combining the knowledge and efforts of many talented programmers. As this system grew and grew, I found it hard to pinpoint flaws and determine what exactly was holding FuzzBot back from being a winner at the tables. This is what kept me back from adding more Strategy modules and designing more complex rules that use more than three linguistic variables for the inputs. Both of these artificial intelligence concepts are certainly not implemented to their fullest potential, and perhaps if a team of programmers were assigned to each, the system could be refined to bring about a more successful system.
Nonetheless, I am tremendously satisfied with the outcome of the project. Regardless of the outcomes of the games it plays, FuzzBot shows that is can definitely compete with other bots designed by professional programmers. For me personally, this is a great achievement. In the world of computer science, I hope that this concept can be explored further.
4. Future Work
One feature that I had originally planned on implementing is the use of a"virgin net." I wanted to have a untrained NeuralNet object for each opponent at the table, which would become trained every time the opponent would make a move, using the data describing the circumstances of the action as the input, and the actual move made as the training value. This would be compared to the output triple produced from the existing trained net. Perhaps the training could take place in a separate thread so the game can progress while the net is learning. I hope to incorporate this soon since I think the opponent modeling scheme would greatly benefit, and in turn, the accuracy of the predictions.
Another feature I wish to enhance is the existing neural net that serves as the core of the opponent modeling system. Right now, there are only eight inputs to the net, and in the future, I would like to build this up as much as reasonably possible. Some other inputs I would incorporate would deal with the results of showdowns. In a showdown, at least one player is required to show their cards at the end of the hand. I would use this information to query the net whenever possible, since this is the most valuable information a player can have abut their opponent – their hole cards!
When I ask my friends who are dedicated poker players about their opinions on my system, the responses I get most commonly deal with how FuzzBot decides its strategy. I have been told that each strategy choice should influence the next, and I am convinced that this could be a key improvement. As of now, it is possible for FuzzBot to bluff every round! Certainly, this is not a good strategy. My initial idea is to incorporate a "decision mask" into the NewStrategy class that would determine, if a given strategy were to be chosen, which ones would be permitted to be chosen in future rounds. For example, if CheckRaise were chosen on the turn, then its mask would prevent FuzzBot from choosing it again on the river, and perhaps the Bluff strategy as well. Of course, this is difficult to implement, since one card can change everything, so this would have to be brought into consideration.
References
[1] Billings, Darse, Computer Poker, http://www.cs.ualberta.ca/~darse/msc-essay/thesis.html, retrieved September 15, 2006.
[2] BioTools Inc., Poker Academy Pro software and Meerkat API, http://www.poker-academy.com/ai.htm, retrieved September 15, 2006.
[3] Bourg, David M., and Seeman, Glenn, AI For Game Developers, Sebastopol, CA: O’Reilly Publishing, 2004.
[4] Braids, Sam, “The Intelligent Guide To Playing Poker”, Towson, Maryland: Intelligent Games Publishing, 2003.
[5] Buckland, Mat, “Programming Game AI by Example”, Plano, Texas: Wordware Publishing, 2005.
[6] Davidson, Aaron,“Using Artificial Neural Networks to Model Opponents in Texas Hold ‘Em”, http://spaz.ca/aaron/poker/nnpoker.pdf, retrieved September 15, 2006
[7] Davidson, Aaron, Billings, Darse, Shaeffer, Jonathan, and Szafron, Duane, “Improved Opponent Modelling in Poker”, Proceedings of the 2000 International Conference on Artificial Intelligence (2000), 1467-1473.
[8] Davidson, Aaron, Billings, Darse, Shaeffer, Jonathan, and Szafron, Duane, “The Challenge of Poker”, Artificial Intelligence Journal, vol 134(1-2), 201-240, 2002.
[9] Heaton, Jeff, “Programming Neural Networks With Java”, http://www.jeffheaton.com/ai/, retrieved September 15, 2006.
[10] Java 1.4.2 language specification: http://java.sun.com/j2se/1.4.2/docs/api/, retrieved September 15, 2006.
[11] Jones, Lee, “Winning Low-Limit Texas Hold ‘Em”, Pittsburgh, PA: ConJelCo LLC, 2000.
[12] Kendall, Graham and Willdig, Mark, An Investigation of an Adaptive Poker Player, http://www.cs.nott.ac.uk/~gxk/papers/ai01poker.pdf, retrieved September 15, 2006
[13] Marrone, Paolo, JOONE Open Source Neural Net Framework and GUI Editor, http://www.jooneworld.com/, retrieved September 15, 2006.
[14] Poker Pro Labs Inc., Poker Pro 2006 software, http://www.pokerpro2006.com/, retrieved September 15, 2006.
[15] Sazonov, Edward, Open Source Fuzzy Inference Engine For Java, http://people.clarkson.edu/~esazonov/FuzzyEngine.htm, retrieved September 15, 2006.
[16] Slansky, David and Malmuth, Mason, “Hold ‘Em Poker for Advanced Players”, Henderson, NV: Two Plus Two Publishing, 2000.
[17] Wikipedia, “John Von Neumann” and “Game Theory”, http://en.wikipedia.org/wiki/Von_Neumann, retrieved September 12, 2006.
Appendix A: The Interface Layer
Appendix A: Source Code (Interface Layer — FuzzBot.java)
import java.util.Random;
import poker.*;
import poker.util.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.event.ItemListener;
import javax.swing.*;
import java.util.List;
import java.io.*;
public class FuzzBot implements Player
{
private static final String ALWAYS_CALL_MODE = "ALWAYS_CALL_MODE";
private int ourSeat; // our seat for the current hand
private Card c1, c2; // our hole cards
private GameInfo gi; // general game information
private PlayerInfo pInfo; // public player info
private Hand hand; // info on the public cards
private Preferences prefs; // the configuration options for this bot
private double betsToCall;
private PokerBrain pBrain = new PokerBrain();
private HandEvaluator he = new HandEvaluator();
private static boolean windowSet = false;
private static int which_Round;
private static GameData gameData = new GameData();
private final static int AHEAD = 0;
private final static int TIED = 1;
private final static int BEHIND = 2;
private static Frame frame;
public static TextArea textArea, t2;
private static JTable myTable;
public FuzzBot()
{
}
public void actionEvent (int pos, int action, int event)
{
PlayerInfo pi = gi.getPlayerInfo(pos);
int playerSeat = gi.getPlayerSeat(pi.getName());
pBrain.update(playerSeat, action, gi, ourSeat, which_Round, c1, c2);
public synchronized void newGame(GameInfo gInfo, Card c1, Card c2, int seat)
{
this.c1 = c1;
this.c2 = c2;
this.ourSeat = seat;
this.gi = gInfo;
which_Round = 0;
resetLabels();
gameData.reset();
debug("newGame called!");
}
public void gameOverEvent()
{
debug(" ----- END HAND ------ " );
which_Round = 0;
}
/**
* A new betting round has started.
*/
public void stageEvent(int stage)
{
debug("=============================================");
debug("*** round " + stage + " ***");
betsToCall = 0;
which_Round = stage;
gameData.newRound(stage);
}
/**
* Get the players next action.
*
*
* @return the player's action
*/
public synchronized int action()
{
gameData.update(gi);
PlayerInfo pi = gameData.gi.getPlayerInfo(gameData.seat);
boolean hasActed = pi.hasActedThisRound();
if (!hasActed)
{
debug("Fuzz for the FIRST TIME");
pBrain.setStrategy(gameData, c1, c2);
}
debug("Getting move...");
return pBrain.getMove(gameData, c1, c2);
}
/**
* Calculate the raw (unweighted) PPot1 and NPot1 of a hand. (Papp 1998, 5.3)
* Does a one-card look ahead.
*
* @param c1 the first hole card
* @param c2 the second hole card
* @param bd the board cards
* @return the ppot (also sets npot not returned)
*/
public double ppot1(Card c1, Card c2, Hand bd) {
double[][] HP = new double[3][3];
double[] HPTotal = new double[3];
int ourrank7, opprank;
int index;
Hand board = new Hand(bd);
int ourrank5 = he.rankHand(c1,c2,bd);
// remove all known cards
Deck d = new Deck();
d.extractCard(c1);
d.extractCard(c2);
d.extractHand(board);
// pick first opponent card
for (int i=d.getTopCardIndex(); i<Deck.NUM_CARDS; i++) {
Card o1 = d.getCard(i);
// pick second opponent card
for (int j=i+1; j<Deck.NUM_CARDS; j++) {
Card o2 = d.getCard(j);
opprank = he.rankHand(o1,o2,bd);
if (ourrank5 > opprank) index = AHEAD;
else if (ourrank5 == opprank) index = TIED;
else index = BEHIND;
HPTotal[index]++;
// tally all possiblities for next board card
for (int k=d.getTopCardIndex(); k<Deck.NUM_CARDS; k++) {
if (i == k || j == k) continue;
board.addCard(d.getCard(k));
ourrank7 = he.rankHand(c1,c2,board);
opprank = he.rankHand(o1,o2,board);
if (ourrank7 > opprank) HP[index][AHEAD]++;
else if (ourrank7 == opprank) HP[index][TIED]++;
else HP[index][BEHIND]++;
board.removeCard();
}
}
} /* end of possible opponent hands */
double ppot = 0, npot = 0;
double den1 = (45*(HPTotal[BEHIND] + (HPTotal[TIED]/2.0)));
double den2 = (45*(HPTotal[AHEAD] + (HPTotal[TIED]/2.0)));
if (den1 > 0) {
ppot = (HP[BEHIND][AHEAD] + (HP[BEHIND][TIED]/2.0) +
(HP[TIED][AHEAD]/2.0)) / (double)den1;
}
if (den2 > 0) {
npot = (HP[AHEAD][BEHIND] + (HP[AHEAD][TIED]/2.0) +
(HP[TIED][BEHIND]/2.0)) / (double)den2;
}
return ppot;
}
public static void debug(String output)
{
gameData.debug(output);
}
// used for the debug console
public void setupDebugWindow()
{
title_Strength = new Label("Hand Strength:");
title_PotOdds = new Label("Pot Odds:");
label_Strength = new Label();
label_PotOdds = new Label();
title_pTriple = new Label("Triple:");
label_pTriple = new Label();
title_Move = new Label("Move: ");
label_Move = new Label();
title_PPOT = new Label("PPOT:");
label_PPOT = new Label();
title_GameData = new Label("GameData");
label_GameData = new Label();
if(windowSet)
frame.dispose();
windowSet = true;
frame = new Frame("Experimental Frame");
frame.addWindowListener(new WindowAdapter() {
public void windowClosing(WindowEvent evt) {
Frame frame = (Frame)evt.getSource();
// Hide the frame
frame.setVisible(false);
// If the frame is no longer needed, call dispose
frame.dispose();
}
});
frame.setBounds(0,0,500,400);
GridBagLayout gridbag = new GridBagLayout();
GridBagConstraints c = new GridBagConstraints();
frame.setLayout(gridbag);
c.fill = GridBagConstraints.BOTH;
c.weightx = 1.0;
gridbag.setConstraints(title_Strength, c);
frame.add(title_Strength, c);
frame.add(label_Strength, c);
frame.add(title_PotOdds, c);
c.gridwidth = GridBagConstraints.REMAINDER;
frame.add(label_PotOdds, c);
textArea = new TextArea("testBot started! \n");
gridbag.setConstraints(label_PotOdds, c);
frame.add(textArea, c);
frame.setVisible(false);
}
private static void resetLabels()
{
label_Strength.setText("");
label_PotOdds.setText("");
label_pTriple.setText("");
label_Move.setText("");
label_PPOT.setText("");
label_GameData.setText("");
}
public static void setTripleConsole(double[] pTriple)
{
label_pTriple.setText(pTriple[0] + " " + pTriple[1] + " " + pTriple[2]);
}
public JPanel getSettingsPanel()
{
JPanel jp = new JPanel();
final JCheckBox acMode = new JCheckBox("Always Call Mode", prefs.getBooleanPreference(ALWAYS_CALL_MODE));
acMode.addItemListener(new ItemListener()
{
//frame.add(textArea);
}
/**
* A showdown has occurred.
* @param pos the position of the player showing
* @param c1 the first hole card shown
* @param c2 the second hole card shown
*/
public void showdownEvent(int seat, Card c1, Card c2)
{
debug("showdownEvent called!");
}
/**
* A new game has been started.
* @param gi the game stat information
*/
/**
* A player has won money.
* @param pos the player position
* @param amount the amount won
* @param handName the name of the winning hand
*/
public void winEvent(int seat, int amount, String handName)
{
debug("winEvent called!");
}
private String getActionString(int move)
{
if (move == 0) return "folds.";
else if (move == 1) return "checks.";
else if (move == 2) return "calls.";
else if (move == 3) return "bets.";
else if (move == 4) return "raises.";
else if (move == 5) return "posts small blinds.";
else if (move == 6) return "posts big blinds.";
else return "????";
}
private static Frame newFrame;
private static Label title_Strength, title_PotOdds;
public static Label label_Strength, label_PotOdds;
private static Label title_pTriple, label_pTriple;
public static Label title_Move, label_Move;
public static Label title_PPOT, label_PPOT;
public static Label title_GameData, label_GameData;
}