Accuracy Enhancement for License Plate Recognition

Lihong Zheng School of Computing and mathematics

Charles Sturt University, Australia lzheng@csu.edu.au

Xiangjian He Faculty of Engineering and IT

University of Technology, Sydney, Australia sean@it.uts.edu.au

Bijan Samali Faculty of Engineering and IT

University of Technology, Sydney, Australia bijan@eng.uts.edu.au

Laurence T. Yang Department of Computer Science, St Francis Xavier

University, Canada ltyang@stfx.ca

Abstract—Automatic License Plate Recognition is useful for real time traffic management and surveillance. License plate recognition usually contains two steps, namely license plate detection/localization and character recognition. Recognizing characters in a license plate is a very difficult task due to poor illumination conditions and rapid motion of vehicles. When using an OCR for character recognition, it is crucial to correctly remove the license plate boundaries after the step for license plate detection. No matter which OCRs are used, the recognition accuracy will be significantly reduced if the boundaries are not properly removed. This paper presents an efficient algorithm for non character area removal. The algorithm is based on the license plates detected using an AdaBoost algorithm. Then it follows the steps of character height estimation, character width estimation, segmentation and block identification. The algorithm is efficient and can be applied in real-time applications. The experiments are performed using OCR software for character recognition. It is shown that much higher recognition accuracy is obtained by gradually removing the license plate boundaries.

Keywords-LPR, image segmentation, blob extraction, CCA

I. INTRODUCTION A license plate is the unique identification of a vehicle.

The fundamental issues in real-time license plate recognition are the accuracy and the recognition speed. License Plate Recognition (LPR) has been applied in numerous applications such as automatically identifying vehicles in a car park, vehicle access control in a restricted area and detecting and verifying stolen vehicles. Till now, there have been some well-known commercially operational LPR systems around the world. These include Safe-T-Cam [1], LPR in USA [2], the ALPR in UK [3], IMPS in Singapore [4], and the CARINA in Hungary [5], etc. Although few details are released to the public about the accuracy of commercially deployed LPR systems, it has been well known that they all work under controlled conditions and the cameras are mounted in fixed locations without mobility.

LPR system consists of two major components: license plate detection and license plate recognition.

License plate detection is the first important step in a LPR system. The quality of a license plate detector influences the accuracy of license plate recognition. On the other hand, many factors can affect the accuracy and efficiency of license place detection. Most of existing license plate detection algorithms are restricted by various controlled conditions such as fixed background [6], known color [7], or designated ranges of the distance between cameras and vehicles [8].

AdaBoost learning algorithm is used as shown in [9] based on both global statistical features and local Haar-features to construct a cascaded classifier for license plate detection. The classifiers based on global features decrease the complexity of detection algorithm. The classifiers based on local Haar-like features further improve the detection rateand reduce the false positive rate. More details are referred to [9].

Among these commercial available systems, two types of classifiers are used. They are OCR-based method and learning-based approach [10]. OCR has shown its advantage in recognizing printed document or text where the background has no or very little noise. However, license plate images captured in real-time usually contain heavy noise and are with complex backgrounds. Therefore, those systems which are claimed to have high accuracy only work well under some restricted constrictions. OCR based recognition requires image pre-processing to remove the boundaries before it can be properly used for license plate recognition under complex and unrestricted conditions. To the best knowledge of authors, there are no papers around which have shown clear, efficient and successful techniques for satisfactorily removing the boundaries of license plates or for accurate segmentation of license plates from their backgrounds.

Learning-based approach such as those shown in [10, 11] extracts optimal features of characters to improve the recognition accuracy. This approach heavily relies on the accurate character segmentation on license plates. However, images taken in real-time can be very difficult for proper character segmentation due to poor image quality.

978-0-7695-4108-2/10 $26.00 © 2010 IEEE

DOI 10.1109/CIT.2010.111

511

2010 10th IEEE International Conference on Computer and Information Technology (CIT 2010)

978-0-7695-4108-2/10 $26.00 © 2010 IEEE

DOI 10.1109/CIT.2010.111

511

2010 10th IEEE International Conference on Computer and Information Technology (CIT 2010)

978-0-7695-4108-2/10 $26.00 © 2010 IEEE

DOI 10.1109/CIT.2010.111

511

Improperly segmented characters will result in misrecognized characters.

In this paper, we present an approach that can efficiently identify the characters in the images of license plates by gradually removing the license plate boundaries. This approach is based on some well-known techniques including image binarization, edge detection, image projections and connected component analysis.

The rest of the paper is organized as follows. In Section II, we give a brief overview of license plate detection and localization. The techniques for boundary removal including color reverse, edge detection, and blob extraction are presented in Section III. Character recognition based on OCRs is proposed in Section IV. Experimental results are demonstrated in Section V. Finally, we conclude this paper in Section VI.

II. LICENSE PLATE DETECTION As shown in [9], the basic idea of the detection algorithm

was to use a variable scanning window moving around on the input vehicle image. At each position, the image area covered by the scanning window was classified using a pre-trained classifier as either a license-plate area (a positive decision) or a non-license-plate area (a negative decision). The classifier used in this algorithm was a significant extension of Viola and Jones’ work shown in [12] to license plate detection.

The algorithm adapted for license plate detection is shown in [9]. We construct a six-layer cascaded classifier to increase the detection speed, in which the first two layers are based on global features and the last four layers are based on local Haar-like features. The classification process can be taken as a degenerate decision tree containing multi-layer classifiers as shown in Figure 1. A positive result from the first classifier triggers the evaluation of a second classifier. A positive result from the second classifier triggers a third classifier, and so on. A negative outcome at any layer leads to the immediate rejection of the image region (block). In other words, those image regions that are not rejected by the initial classifier will be processed by a sequence of classifiers. If any classifier rejects a selected image region, no further processing will be performed for this region. It is commonly seen that, for a given vehicle image, the majority of evaluated image regions are negative. Therefore, the cascaded classifier shown in Figure 1 attempts to reject as many negatives as possible at the earlier stages. As its consequence, this cascaded classifier leads to fast license plate detection. It is worth to note that the detector (classifier) acts on the vertical edge maps of input vehicle images rather than on the raw image intensity values. This further enhances the efficiency of the cascaded classifier.

Figure 1: Process of constructing a cascaded classifier [9].

The reason of using the above mentioned method for detection before recognition is that the algorithm can be achieved in real-time and accurately, and can be applied for more flexible applications under complex environmental conditions.

Figure 2 shows the examples of a few original license plates areas extracted. The located license plates are marked in red boxes.

Figure 2: Images showing the license plates extracted in red boxes.

III. CHARACTER SEGMENTATION OF LICENSE PLATES Once the license plate area is found, the following step is

to find the characters contained in the image. The procedures of character segmentation include three different tasks. The first one is character height estimation. The upper and lower boundaries of character are located and used to obtain the character height. Then, we move to estimate character width and segment license plate. The final step is to construct the labeled character segments based on block extraction algorithm for verification of character segments.

A. Character Height Estimation of License Plates This step contains three parts: color reverse, vertical edge

detection and horizontal projection histogram 1) Color reverse

The license plates in NSW, Australia have many different formats, colors, and alignments. For instance, while in black, black in white, black in yellow, etc. are commonly seen. The characters in located grey level image are interest areas in identifying the license plate. Color reserve step is necessary before we produce the binary image containing characters. It converts the color of the characters on a license plate to black. It is done based on statistical analysis of edges. Given an (m × n) sized image, we pick l horizontial symmetrical lines of the image. The color index is calculated as the average amount of the cross points along each line in horizontal direction. Cross point number is increased by one if there is a foreground point.

),(EdgePoint1ColorIndex,∑=

jiijx

l

where j is from 1 to l and i is from 1 to n. If ColorIndex value is over a preset threshold, the candidate image is labeled as an image to be converted. Otherwise, keep the original value for next step.

Therefore, all candidate license plates are assumed to be black-on-white, and the consistency of ROI is kept.

512512512

2) Vertical edge detection Once a license plate area is detected and located, a

rectangular area of the license plate obtained by using the algorithm described in Section II may not cover the whole license plate. However, it is expected that the area will fully contain the license plate. Figure 3 shows the located areas of the license plate images in Figure 2.

Figure 3: Images from those in Figure 2.

Given that the located area shows stronger connectivity

in vertical direction than horizontal direction, we perform vertical edge detection on the license plate images as shown in Figure 3 through the computation of horizontal gradient. At each pixel, use the Sobel mask of [-3 0 3;-10 0 10;-3 0 3] to compute the horizontal gradient value [13]. Then, use the Otsu [14] method for binarization to obtain the vertical maps. More about the Otsu method will be addressed in Subsection B 1). The edge pixels are represented using white pixels and other pixels using black pixels. Figure 4(a) shows the vertical edge pixels on the images in Figure 3.

3) Horizontal projection histogram

(a) (b)

Figure 4: (a) Vertical edge maps of images in Figure 3; (b) Horizontal projection histogram.

The use of projection histogram is not a new concept. It is used here to find the upper and lower bounds of a license

plate after the vertical edge are obtained. We perform a horizontal projection to find the top and bottom position of characters. The value of a histogram bin is the sum of the white pixels along a particular line in horizontal direction. When all values along all lines in the horizontal direction are computed, the horizontal projection histogram is obtained. The mean value of the histogram is then used as a threshold to decide the upper and lower bounds. The middle area of which the histogram segment is bigger than the threshold is recorded as the area bounded by the upper and lower bounds. The horizontal projection histograms of the six images shown in Figure 4(a) are displayed in Figure 4(b).

Hence, the distance between upper and lower boundaries is recorded as the height value of characters.

B. Character Width Estimation of License Plates After the area bounded by the upper and lower bounds of

a license plate is found, the areas above the upper bound and below the lower bound are removed. The remaining area without the upper and lower boundaries of the license plate is considered for character segmentation on the license plate. Image binarization and vertical projection are two steps for segmentation here. The purpose of segmentation is not the same as the one for the learning-based approach. Each segment here may or may not contain a character. We use segmentation results to estimate the width of the characters on the license plate for further processes. Each segment does not need to be accurate to contain exactly one character.

1) Image binarization As well known, image binarization is to change grey

values of an image into binary values and re-represent the image as a binary image accordingly. Image binarization highlights the pixels of interest and suppresses the background pixels. The simplest way for image binarization is to choose a threshold value, and classify all pixels with values above this threshold as white (255 grey value), and all other pixels as black (0 grey value). Otsu [14] gave an idea to select a good threshold globally. Otsu’s method is based on an analysis of the gray scale level histogram of the whole image and selects an optimal threshold for a given image by maximizing a discriminant criterion, i.e., the separability of the resultant classes in gray levels. Figure 5 shows the results of binarization on the images in Figure 3 after cutting the upper and lower boundaries of the license plates.

Figure 5: Binarized images of the images in Figure 3 with upper and lower

bounds removed.

2) Vertical projection histogram We perform a vertical projection to find the gaps between

characters on a license plate. The value of a histogram bin is the sum of the white pixels along a line in vertical direction. When all values along all lines in the vertical direction are computed, the vertical projection histogram is obtained. Figure 6 shows the vertical projection of images in Figure 5.

513513513

Figure 6: Vertical projection of the images in Figure 5.

Figure 7: Character Segments of license plates.

Based on the results of vertical projection, each license

plate is separated into blocks horizontally by the zero points in the projection histogram. Figure 7 shows the segments (or blocks) of the images in Figure 3 obtained from the results of vertical projection after upper and lower bounds removed.

To estimate the width of a character after the segmentation process above, we take into account only the blocks with widths bigger than 6 pixels and smaller than the widest block. The averaged widths of these blocks are used as the estimated width of characters.

The estimated height and width of characters is used in the following step for blob checking as part of checking criteria.

C. Distinguishing Character Blocks by Blob Extraction As shown in Figure 7, not all the segments obtained in

Subsection B are blocks containing characters. Some blocks actually contain nothing but vehicle background and/or left and right bounds of the license plates. The statistical features of such background are very similar to that of the characters. The previous features that we have got are not good enough to tell between background and characters. Roughly, 70% character blocks can be identified successfully. Therefore, other help is sought to improve the successful rate. It only keeps the blocks containing characters and removes those blocks that do not contain characters. By doing this, blob extraction algorithm is applied for tracking character blocks in the left and the right side. The left and right boundaries of the license plates will also be located and removed. This step consists of two parts: blob detection and blob checking.

1) Blob detection algorithm Blob extraction [15] is another kind of image

segmentation techniques that labels the pixels in an image as belonging to one of many different groups. It is a sort of extension method based on Connected Component Analysis (CCA) algorithm [16]. Blob extraction is limited on the binary image in our system only. It includes region labeling, connected-component labeling, blob discovery and region extraction. Blobs can be counted, filtered and tracked.

Well-known algorithms for blob extraction can be sorted into two types: the pixel-based algorithms [15] and the run based algorithms [17]. Given an image, they both assign

labels to a pixel such that adjacent pixels of the same features are assigned the same label. The image is copied into a small or large array or arrays. The pixel-based algorithms do multiple scans. On the other hand, the run based method only search connected pixels in a run. In this paper, the first approach is applied in blob finding process. The brief idea is described as follows.

First, a label counter is created and initialized to one. Then, the binary image is scanned from left to right and from top to bottom.

• For every pixel, check the northeast, north, northwest, and west pixel for 8-connectivity for a given region criterion (i.e., intensity value of 1 in binary image). • If none of the neighbors fits the criterion, then assign to

region the value of the region counter. Increase the region counter by one.

• If only one neighbor fits the criterion, assign the pixel to that region. If multiple neighbors match and are all members of the same region, assign the pixel to their region.

• If multiple neighbors match and are members of different regions, assign the pixel to one of the regions (it doesn't matter which one). Record all these regions as ‘equivalent regions’.

• Scan image again, and assign all equivalent regions the same region value.

• Scan stops until there are no unlabelled pixels in the image.

Figure 8 below shows the extracted blocks in red boxes from the located license plates.

Figure 8: Extracted blocks from images in Figure 3.

2) Blob checking

Next, we move to identify the blocks based on character block checking criteria, so that the non-character blobs can be removed after the process. There are many different conditions that can be selected as the checking criteria. The example of the conditions are the upper and lower boundary positions, the blob area, the blob perimeter, mean, density, the estimated character width and height as shown in Subsections A and B, the height to width ratio, and maximum characters amount in one license plate. Most of these can be included in the character block checking criteria.

Following are some examples of the criteria. Rule 1: if the upper boundary of one blob is smaller than

the overall UPBound, or if its upper boundary is bigger than the overall LOWBound, the candidate blob is a non-character one.

Rule 2: if the height to width ratio of a blob is bigger than 0.8 or smaller than 0.3, the candidate blob is assumed to be a non-character one.

514514514

Rule 3: if the area of blob is bigger than one sixth of the whole image size, it needs further segmentation.

Rule 4: if the amounts of successful blocks are more than 6 or less than 3, the candidate needs to be re-checked.

The blob recorded as a possible character block must match all the conditions to be recorded as a character block and retained. Otherwise, the blob is regarded as a non-character block and hence is removed. After all blocks on the given license plate are checked, the process completes. Then, the selected blobs are grouped together and go through image binarization again to remove the unwanted non-character areas (NCRs) of the license plate.

Figure 9: The binary and enlarged license plate images

IV. LICENSE PLATE RECOGNITION For character recognition using an OCR, the following

steps are performed. • Binarize the images with non-character are

removed. The binarization process repeats using the Otsu method.

• If a binarized image has white characters and black background, inverse the pixel values so that the characters become black and background becomes white. This is done simply by retrieving the flag value obtained in the early stage.

• Enlarge the binary images obtained in Steps 1 and 2 by evenly adding white pixels in all four directions: left, right, up and down so that the enlarged images are 50% wider and higher. This is to guarantee enough page margins for the OCR to read all characters on the binary images obtained in Step 2.

• Apply an OCR to the enlarged images obtained in Step 3 for license plate recognition.

As an illustration, for the images in Figure 3, the binary and enlarged images are shown in Figure 9 above.

V. EXPERIMENTAL RESULTS Experiments have been performed to test the proposed

system and to measure the improved accuracy of the system. The system is designed in VC 6.0 for recognition on Australia license plates. The test image frames were taken from real time video footages under various illumination conditions The input image frames are gray level images with the size of 640×480.

The license plate detection approach presented in Section II was used to obtain license plate images whose size is 136×45. Totally, 3549 frames which contained 594 different license plates were included in the experiment. All candidate images are processed in one format, i.e., black characters on white plate. The binary enlarged license plates images are

obtained by our proposed method and sent to the ABBYY OCR software [18] for recognition.

It is shown that accuracy for the extraction of plate region is 96.4% and 91.0% for the segmentation of the characters and 98.7% is the percentage of accuracy of the recognition unit. The overall system performance can be defined as the product of all units accuracy rates (extraction of plate region, segmentation of characters and recognition of characters).

Recognition Rate of LPR System = 86.7 % (Percentage of Accuracy)

The overall accuracy combining detection and recognition is about 86.7%. The results of the tests are given by Table I.

As a comparison, the sample license plates are being recognized using ABBYY without going through the process of non-character area removal shows only 30.6% recognition rate together with 44% false positive rate (false alarm).

We see the results before and after the non character area removal as shown in Table 1. It proves that a very high recognition rate of license plates can be achieved when the character of license plates are properly segmented using the algorithm proposed in this paper

TABLE I. TABLE 1: RECOGNITION RESULTS USING OCR SOFTWARE

Accuracy Rate of License Plates

Before NCR Removal

After NCR Removal

Detection Rate of License Plates

96.4% 96.4%

Accuracy of Character Segmentation

NA 91.0%

Character Recognition Rate

30.6% 98.7%

Overall Recognition Rate 29.5% 86.7%

VI. CONCLUSIONS In this paper, we have proposed a method to segment the

characters of car license plates. It removes all unwanted areas. This is a crucial work after the detection of license plates and before the use of OCRs for character recognition. The process is successful through the steps of character vertical height estimation, character width estimation, and segmentation of a license plate into blocks and the identifying of character blocks. The techniques used include image binarization, vertical edge detection, horizontal and vertical image projections and blob extraction. Various well-known techniques are applied to come out with the innovative algorithm in this paper.

The ABBYY OCR software was used to test our results. The experimental results show a high accuracy of non-character area removal and significantly higher recognition rate after character is segmented. The recognition has increased from about 30.6% before our proposed process to

515515515

86.7% after we remove all unwanted background areas. Therefore, the recognition accuracy has been enhanced

REFERENCES [1] http://vision.cmit.csiro.au/project/stc/ [2] http://www.perceptics.com/license-plate-reader.html. [3] http://www.ivsuk.com/anpr.asp. [4] http://www.singaporegateway.com/optasia/imps. [5] http://www.arhungary.hu/. [6] H. Bai and C. Liu. A Hybrid License Plate Extraction Method Based

on Edge Statistics and Morphology. in Proceedings of 17th International Conference on Pattern Recognition. 2004.

[7] S. K. Kim, D.W. Kim, and H.J. Kim. A Recognition of Vehicle License Plate Using a Genetic Algorithm Based Segmentation. in Proceedings of International Conference on Image Processing. 1996.

[8] S. Kim et al. A Robust License-plate Extraction Method under Complex Image Conditions. in Proceedings of 16th International Conference on Pattern Recognition. 2002.

[9] H. Zhang, W. Jia, X. He, Q. Wu, Learning-Based License Plate Detection in Vehicle Image Database, International Journal of Intelligent Information and Database Systems (IJIIDS), ISSN: 1751-5858, Inderscience, Vol. 1, No. 2, 2007, pp.228-243.

[10] Y. Zheng, H. Li and D. Doermann, Machine printed text and handwriting identification in noisy document images, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, pp. 337 - 353, Mar. 2004.

[11] B. Zhao, Y. Liu, and S.-W. Xia, Support vector machine and its application in handwritten numeral recognition, Pattern Recognition, 2000. Proceedings. 15th International Conference on, vol. 2, pp. 720 - 723, Sept 2000.

[12] P. Viola and M.J. Jones, Robust Real-Time Face Detection. International Journal of Computer Vision, 2004. 57(2): pp. 137-154.

[13] http://www.pages.drexel.edu/~weg22/edge.html. [14] N. Otsu, A threshold selection method from gray-level histograms,

IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, pp. 62-66, 1979.

[15] Horn, Berthold Klaus Paul (1986). Robot Vision. MIT Press. pp. 69–71. ISBN 0-262-08159-8.

[16] Michael B. Dillencourt and Hannan Samet and Markku Tamminen (1992). "A general approach to connected-component labeling for arbitrary image representations}". J. ACM. http://doi.acm.org/10.1145/128749.128750.

[17] Lifeng He, Yuyan Chao, Kenji Suzuki, Hidenori Itoh: A Run-Based One-Scan Labeling Algorithm. ICIAR 2009: 93-102

[18] http://www.abbyy.com/

516516516