“Agricola” Board Game Assist Program

EGGN 512 – Computer VisionDerek LangMay 7, 2012

IntroductionGerman-style board games are a class of board games that usually rely on strategy, de-

emphasize luck and conflict, and keep all players in play until the end of the game [1]. Typically, the rules are simple to keep the game accessible for players of different ages and nationalities. One notable exception is the game Agricola. There are 360 cards, and 17 different types of tokens (to a total of 317 total) that are used for the game. The game's concept is that of farm management in late 17th century Europe. A player's board represents their farm. The score is largely based on what tokens are present on their farm.

The purpose of this paper is to provide a means of processing an image of a player's end-game board, and to compute a score. Hopefully, this algorithm will provide a quick means of calculating the score and determining the winner of the game.

Color images are taken using a DSLR camera as JPEGs under the same lighting conditions. 17 images were used (see Fig. 1), 1 for calibrating the board detection, 16 of varying end-game conditions. The images were then rectified using the Bibble® software lens correction plugin. All code was developed in Matlab as a proof of concept.

This paper will first discuss the existing literature relevant to this project. It will then describe in detail the techniques used in this project. The project results will be shown. Finally, a discussion of the results and recommendations on future work will be presented along with reference material. An appendix of all the code developed for this project is included at the end of the paper.

Related WorkMarkers and recognizable patterns have been used extensively to determine the pose of objects

in relation to a camera. One common example is the concentric contrasting circle (CCC) [2]. A CCC consists of a white circle centered in a black circle. An intensity threshold can be taken of the image, and correspondence between the centroids of the circles will determine a CCC detection. However, using multiple black and white CCCs will need a non-trivial solution to determine the geometry of the points, so colored CCCs will be used.

Color segmentation is usually considered where the colors and lighting conditions may not be known in advance. One method by Wang and Watada [3] uses the Karhunen-Loeve transform to extract the most information from a color image, into a 2-dimensional form, and to use an Otsu [4] threshold to segment. Another method would use K-means clustering [5] to determine which colors to threshold around. For this project, the colors of the board are known in advance, so a simple threshold can be determined through experimentation.

The SIFT algorithm [6] is a widely used method for creating and matching feature descriptors. In SIFT, a keypoint is detected from a LOG scale-space, and a feature vector is created from the neighborhood gradient around that keypoint. Ke and Sukthankar [7] developed an optimization of the SIFT algorithm. The algorithm alters the last step of the SIFT algorithm (keypoint description) and instead of building the descriptors from the local image gradients of the image, a precomputed eigenspace is used to express the gradient images, which is used to project a keypoint's local image gradient, deriving a feature vector that only contains the top neigenvectors for a given image gradient.

Calonder, Lepetit, Strecha, and Fua [8] also developed a feature descriptor, which they called a 'Binary Robust Independent Elementary Feature' (BRIEF). An image patch surrounding a keypoint can be described as a binary string of comparisons between different regions of the image patch. The descriptor is very fast to compute, and the Hamming distance can be used to match features, which is faster and more efficient than the L2 norm used in other feature descriptors.

AlgorithmThe main algorithm can be described in the following steps:

1. Detection of the board

2. Detection of tiles

3. Detection of player pieces:

◦ Persons

◦ Stables

◦ Fences

4. Detection of livestock pieces

5. Detection of crop pieces

6. Detection of pastures

7. Calculation of score

Although there are a lot of different pieces to consider for this project, there are many factors that help to simplify the task. Knowledge of the rules of the board can be used as logic that will reduce the amount of detections that have to be made and help check for errors. Once the detections have been made, a score can be calculated using lookup tables.

Board Detection

The board is detected by placing four color markers similar to CCCs [1] at the four corners of the board. The image is converted to HSI values, and a narrow band hue threshold of the image can be taken for each of the targets. The white marker centers can be found by taking a threshold of the intensity of the image. The markers are then found by matching the centroid of the

Fig. 1: Samples of the test images.

colored circle to the centroid of the white center. Finding the four markers will give a correspondence to the known geometry of the board, which can be used to create a transform to convert the image into a orthornormal perspective. See Fig. 4 for an example of this normalization.

The normalized test image can be further subdivided into 54 regions that can be categorized as follows: 15 squares where most of the pieces are placed, the 38 bordering regions where the fence pieces are placed, and a region at the top of the board where a 'stockpile' of extra pieces for the player is placed. This reduces the complexity of detection algorithms, as fewer pieces will have to be detected at once, and smaller images to process will be faster to process.

Tile Detection

The tiles that are placed on the board are of four different types: Wooden house, Clay house, Stone house, and Field. The SIFT algorithm with a coarse Hough space [9] is used to determine a match between the template and the test region. Since there are 16 different images used for each of the tile types in the game, it is possible that 3,456 SIFT comparisons will be made. A few simplifications are made to increase the likelihood of matching and reduce the number of comparisons:

1. A full transformation between the matching sets of points is not calculated, only the difference between the descriptors is used as a rough transformation.

2. The angle of the transformation should be close to 90 degrees, so transforms outside of this range can be discarded.

3. Any transformation that does not have a minimum number of points is discarded.

4. From the game rules, it is known that your house will only be built of one type of material. A check for one type of house is skipped if another type is found.

5. If a match is found for one region, that region is not used for subsequent comparisons.

Player Piece Detection

There are three types of pieces that are specific to a player on a board. For simplification, only

Fig. 2: Test image (left) and orthonormal transformed image (right)

one color is considered for this project. The 'person' and 'stable' pieces are considered using the following method: On a region, a hue threshold is performed to isolate the piece from the background. The binary threshold is then morphologically closed and eroded to clean up the detection. From this binary threshold, the area and the centroid of the isolated piece can be used to determine a match for that piece. Since normalization can distort a piece to 'expand' into the neighboring regions, The centroid of a piece will lie within a radius of the center of the region (see Fig. 3).

The 'fence' pieces are isolated using a hue threshold, but instead of calculating the regional properties for that binary image, a simple percentage is calculated from the binary image. If the percentage of blue pixels to the total region is above a threshold (20%) then there is a match for a 'fence' piece in that region.

Livestock Piece Detection

There are three type of livestock pieces: the sheep (white), the boar (black) and the cattle (brown). Since they are all the same cube shape and occur in the same regions, they are given the same consideration. We only check the regions that do not have a field tile, since the game rules precludes this from occuring. Thresholds based on HSI conversion of the region are made to produce a binary detection. Hue and saturation are used for the cattle, and saturation and intensity are used for the sheep and boar. Morphological operations are used to isolate a detection from noise. The binary images' area is then compared to a threshold to ensure a detection.

Crop Piece Detection

There are two types of crop pieces: the grain (yellow) piece and the vegetable (orange) piece. The method to detect them is similar to detecting player or livestock pieces. A hue threshold is performed on a region, and the region properties of 'blobs' are thresholded to ensure detection. We use knowledge of the game's rules to only check for pieces on the large top region, and on top of filed tiles. Since the pieces are rotationally invariant discs, we can use a much larger erosion element to isolate each piece (see Fig. 4).

Fig. 3: Player Piece detection. RGB image (left) and corresponding hue threshold (right). Note that the centroid for the fence from the neighboring region is much closer to the edge than the true detection.

Pasture Detection

Pastures are defined as any squares that are enclosed by fences on all sides. Since these can be of an arbitrary shape and size, a hole-filling algorithm [10] is used. A representational 7 by 11 pixel binary image of the board is created with all the fence detections used to construct outlines of the pastures. Using Matlab's imfill function, the filled representational image is xor'd with the original representational image, to create an image that contains only the detected pastures. It is then a simple matter to count pastures for detection. (see Fig. 5)

Results

Board Detection and orthonormal image creation

The board detection algorithm took 19 seconds on average to run. Marker detection was successful on all but one image (9515.jpg), making it 93.75% accurate. It is believed that glare on the board from nearby light over-saturated the image, affecting the threshold of the image (see Fig. 6).

Fig. 4: Crop piece detection. The hue threshold is eroded to isolate the pieces from possible touching.

Fig. 5: Pasture detection. Representational image showing fences (left) and resulting pastures (right).

The calculated scores versus their actual scores is shown in Table 1. This method is 83.65% accurate based on these testing images. It is interesting to note that there were no false positives in detecting pieces, only false negatives. A SIFT mismatch would count one tile as another type, and that detection would cause checks for the other tile types to be skipped (see Fig. 7). A false positive would only occur if a person piece was not covering the center of the tile. It may be possible to avoid this detection by obscuring the center of the template images, but it is unknown if this would affect the robustness of the detection.

Image Calculated Score Actual Score Accuracy Notes9512 10 11 90.91% Missed sheep piece9513 17 18 94.44% Missed cattle pieces9514 10 21 47.62% SIFT mismatch9515 0 22 0.00% Marker detect failure9516 32 33 96.97% Missed cattle pieces9517 23 24 95.83% Missed cattle pieces9518 29 29 100.00%9519 25 26 96.15% Missed stable9520 28 28 100.00%9522 12 12 100.00%9523 9 16 56.25% Missed fence9524 19 23 82.61% Missed fence9525 22 23 95.65% Missed fence9526 24 25 96.00% Missed fence9527 49 50 98.00% Missed fence9530 22 25 88.00% 2 persons stacked

Table 1: Scoring results.

Fig. 6: Marker detection miss. Successfully detected marker (left) and unsuccessfully detected marker (right)

A fence piece would be missed if the piece was not completely in the region where it would be checked (see Fig. 8). An attempt to mitigate this was to enlarge the regions where they would be checked, but a miss like this could be attributed to player error of the piece not being in the correct region.

Livestock pieces would be missed if the pieces were touching (see Fig. 9). The threshold detection would treat the pieces as one large 'blob', and only one would be counted for multiple pieces. Many attempts were tried to remedy this, such as checking the area or bounding rectangle of a detection. This was exacerbated by occlusion of the piece and distortion of the piece due to the orthonormal transformation, which would lead to greater variance in the detection.

Fig. 9: Livestock miss. Touching pieces are not isolated from eachother in a hue threshold.

Fig. 8: Fence miss. Note that the borders of the square above and below the region are visible.

Fig. 7: False positive tile detection.

DiscussionThis method for calculating the score is largely accurate, with usually only one miss to account

for the difference in score. The score in 9 of the 16 cases was also only off by one point.

Errors in fence piece registration could be mitigated by further enlarging the region where the detection is checked. There is a condition where in the game, a player can stack two person pieces on top of each other. For the sake of simplicity of this method, this condition was not considered. The biggest obstacle to in correcting the score would be accounting for the cases where livestock pieces were touching. One way to reduce the exacerbating effects of the normalization of the board would be to require the picture to be taken from top-down. It would then be much easier to use the area of a detection to determine if there were multiple pieces in a single 'blob'.

The time to run the algorithm would vary between 100 to 200 seconds. The main contributing factors to this length are the board marker detection and the SIFT tile detection. This could be improved by a number of means. Marker detection relies on comparing the centroid of the colored markers to their white centers, which can result in thousands of region property comparisons being made. This can be reduced by using solid markers, such as in Hislop's project [11], or by a single marker (similar to an AR target) that could give the pose needed to determine a transformation. The SIFT feature descriptor could be replaced by a more compact alternative, such as a BRIEF [8] descriptor, PC-SIFT [7], or SURF [12].

Before the development of this algorithm, a computer spreadsheet was created to help in calculation of the score. The time to determine score (for each player) is comparable or shorter to this algorithm. If this algorithm were optimized, robustified and ported to a stand-alone program, it could be preferable to use this as a means of determining score.

References[1] Web. “German-Style board game”, http://en.wikipedia.org/wiki/German-style_board_game, accessed May 7, 2012.[2] L. Gatrell, W. Hoff, C. and Sklair, “Robust Image Features: Concentric Contrasting Circles and Their Image Extraction,” Proc. of Cooperative Intelligent Robotics in Space, Vol. 1612, SPIE, W. Stoney (ed.), 1991. [3] C. Wang, and J. Watada “Robust Color Image Segmentation by Karhunen-Loeve Transform based Otsu Multi-thresholding and K-means Clustering”, 2011 Fifth International Conference on Genetic and Evolutionary Computing, pp. 377-380, 2011.[4] N.Otsu, “A threshold selection from gray level histograms”, IEEE Trans. Systems, Man and Cybernetics, vol.9, pp. 62-66,Mar. 1979.[5] R. Szeliski, Computer Vision: Algorithms and Applications, Springer International, pp. 289-292, 2010.[6] R. Szeliski, Computer Vision: Algorithms and Applications, Springer International, p. 223, 2010[7] Y. Ke and R. Sukthankar, “PCA-SIFT: A More Distinctive Representation for Local Image Descriptors ,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’2004), pp. 506–513, 2004.[8] M. Calonder, V. Lepetit, C. Strecha, and P. Fua, “BRIEF: Binary Robust Independent Elementary Features,” in Proceedings of the European Conference on Computer Vision, 2010.

[9] W. Hoff, “EGGN 512: Computer Vision”, http://egdegrees.mines.edu/course/eggn512/lectures/, Colorado School of Mines, USA, accessed May 7, 2012[10] P. Soille, Morphological Image Analysis: Principles and Applications, Springer-Verlag, pp. 173-174, 1999.[11] E. Hislop, “Augmented Reality for a Board Game”, Colorado School of Mines, W. Hoff (prof.), 2011.[12] H. Bay, A. Ess, T. Tuytelaars, L. V. Gool, “Surf: Speeded Up Robust Features”, Computer Vision and Image Understanding 10, pp. 346–359, 2008.

Appendix A: codefunction score=scoring(fname)% score=scoring(fname)% Agricola board game AR project% main scoring function.% reads fname, and hopefully computes the score corrresponding to that% board.% please make sure that the board has markers, and that the image is% treated properly.p=path;if strfind('vlfeat',p)==[] run('..\vlfeat-0.9.14\toolbox\vl_setup.m') % setup vlfeat toolboxendclear p%% read file% fname=input('enter file name:\n','s');I=imread(fname);%% detect board markers and normalise[Inorm]=board_norm(I);%crop image for processing.crops=board_crop(Inorm);pause(5)%% Detect tilestemplate_structcrops=sift_test(crops,stone);crops=sift_test(crops,clay);crops=sift_test(crops,wood);crops=sift_test(crops,field);%% Detect Stablescrops=stable_test(crops);%% Detect Fences & Pasturescrops=fence_test(crops);crops=pasture_test(crops);%% Detect Personscrops=person_test(crops);%% Detect Livestockcrops=cow_test2(crops);crops=sheep_test(crops);crops=boar_test(crops);%% Detect Vegetables/Wheat (Todo)crops=wheat_test(crops);crops=veg_test(crops);%% Determine numbersrun score_listsquarelist=[8:2:16 24:2:32 40:2:48];% arraysawheat=[crops.wheat];aveg=[crops.veg];