Barcode Fingerprinting: Individual Identification ofCommercial Products Using Print Irregularity in Barcodes

Rina Ueno (Student number: 81724141)Supervisor: Dr. Jin Mitsugi

Sub Supervisors: Dr. Jun Murai, Dr. Osamu Nakamura

I. BACKGROUND

Barcode is a widely used technique to automatically identifyproducts. It is a symbol that encodes data into a machine-readable pattern, varying width, parallel, rectangular dark barsand pale spaces. In particular, every consumer product in theworld is printed a product-level barcode which is an encodingproduct number. Product numbers are distinctive for eachbrand of commercial products. For individual identificationof products, attaching new serial numbers is necessary. Atraditional way to identify individual products is attaching newsymbols like seals or RFID tags which have serial numbers.However, this way is not practical to manage cheap andconsumptive commercial products since it increases productmanufacturing cost and there are few merits for manufactures.In addition, manufactures will feel inconvenient by changingpackage designs which results in changing impressions ofproducts. There is demand for an individual identificationtechnique of commercial products without attaching seals ortags.

II. PROPOSAL

This research proposes identification using print irregular-ities in barcodes referred to as Barcode Fingerprinting, andFigure 1 shows that print irregularities in a product-levelbarcode are so subtle that it is difficult to observe with humaneyes, but so distinct to observe with computer vision. Printirregularities are not deliberate and not printed with specialprinting methods. Even if product-level barcodes presents oneproduct number, print irregularity patterns in the barcodesare different. The proposal enables individual identification ofcommercial products without enforcing explicit serializationbut only with existing product-level barcodes.

III. RELATED WORKS

Soborski proposes a general framework to detect and encodefeatures of print irregularities in marked symbols as patentapplication [1], however, this work doesn’t introduce detailedmethod of them. Takahashi proposes a method to extractfeatures from pearskin patterns in metal surfaces [2] andthis work applies an algorithm which extract features oflocal gradients in images. Wayman presents a key assumptionof all biometric authentication systems is that (i) extractingcharacteristics which are both distinctive between individualsand repeatable over time for same individuals and (ii) finding

feature sets matching the extracted feature set from storedreference feature sets [3].

IV. OBJECTIVES

Main objectives of this research are the followings.• Development of individual identification of commercial

products using print irregularities in existing product-level barcodes.

• Quantitative evaluation of the performance of the identi-fication.

The proposal identifies print irregularity patterns in bar-codes. To identity graphical patterns, it should perform thefeature extraction and the feature matching as same as otherindividual identifications. In addition, it is desirable that Bar-code Fingerprinting can perform product-level identificationand individual identification concurrently. The followings arekey points to develop the identification.

• The feature extraction, which is distinctive and repeat-able, of print irregularities in existing barcodes.

• The pattern matching of the features with those of refer-ence images.

• The concurrent processing of product-level identificationand individual identification.

V. BARCODE FINGERPRINTING

The individual identification of print irregularities in bar-codes is realized by extracting distinctive and repeatable fea-tures of print irregularities and matching the extracted featureswith that of stored reference images. To extract features ofprint irregularities in barcodes, Speed-Up Robustness Features

Fig. 1. Print irregularities in a product-level barcode

Fig. 2. Overview of Barcode Fingerprinting

(SURF) algorithm is chosen and used. For feature match-ing, the method transforms images geometrically and detectsmatching features which are similar in locations and values.

Figure 2 shows an overview of image processes to extractand match features between a query image and a referenceimage. The method processes geometric transformation, fea-ture extraction, and feature matching in order. Geometrictransformation is used to transform 3-dimensional shapes inimages to 2-dimensional surfaces. Since locations, orienta-tions, scales, and shapes of 3-dimensional objects in imagesare different because of camera angle, geometrical transforma-tion is effective to adjust objects in images. SURF algorithmextracts changes of color gradients as features. It extractscharacteristic pixels in images blurred with various blurringvalues as keypoints, and it stores coordinates of the keypointsand the blurring values when it detect the keypoints as scales.It also computes gradient vectors around extracted keypointsand it computes orientations of the keypoints from the vectors.SURF descriptor presents gradient vectors, a feature orienta-tion, and also the scale. Then SURF descriptor is scale- androtation- invariant. The proposal compares extracted featurepoints between barcode images in coordinates and gradientvectors, and detects features whose differences are lower thanpredefined threshold.

VI. EVALUATION WITH PRELIMINARY EXPERIMENT

A. Method

The proposed individual identification method are evalu-ated experimentally by implementing an experimental system.Table I shows an environement of the system. The systemperforms affine transform using feature based transformationin the geometric transform phase. The transformation is basedon correspondences of matching features which are similarin gradient vectors between two images. The features areextracted by SURF algorithm. The system processes barcodeimages taken from 100 plastic bottle products of single brand.The images were taken from upper left of the barcodes. Inorder to take clear outlines of barcodes, a microscope camera

takes the images. Table II shows a specification of the camera.Each barcode was taken twice for both a stored referenceand an input query. The reference images were taken so thatbarcode symbols are perpendicular, and the query images weretaken with intentionally rotating the camera along the yaw axis(perpendicular to the barcode symbol) from the orientationswhere the taken reference images. The proposal method iscompared with a general feature matching method, referred toas Method A. It detects matching features which are similarin gradient vectors. To evaluate results of matching features,numbers of matching features are converted to a matching rateas the following equation.

Matching rate =nmatch

min(nquery, nreference)(1)

To describe the performance of the identification, a confusionmatrix is assigned. True-Positives (TP) shows cases whenimages are taken from a single barcode and the system predictsthe images are taken from a single barcode. False-Negatives(FN) shows cases when images are taken from a single barcodebut the system predicts the images are taken from differentbarcodes. Recall rate is a rate how often predictions are correctwhen images in the Same cases. The recall rate is calculatedby the following equation.

Recall rate =TP

(TP + FN)× 100 (2)

B. Result

The system computed 10,000 patterns of feature matchingbetween 100 template images and 100 input images. Figure3a shows the probability distribution of matching rates withMethod A and Figure 3b shows that with the proposal. Thehorizontal line shows the matching rates and the left horizontalone shows the frequency when images are taken from twobarcodes, and the right horizontal one shows the frequencywhen images are taken from single barcode. Blue bar showshistogram when images are taken from two barcodes andred one shows histogram when images are taken from singlebarcode.

C. Consideration

This experiment assumes that the identification system isset a threshold of the matching rate to identify query images

TABLE IENVIRONMENT OF THE PRELIMINARY EXPERIMENTAL SYSTEM

OS Windows 10Programming Language MATLAB R2017a

Libraries MATLAB Image Processing Toolbox

TABLE IISPECIFICATION OF THE MICROSCOPE CAMERA (MIYOSHI, UK-02)

Camera sensor CMOS 200 million pixelMagnification About 20 timesFocus length Manual 10mm - 500mm

(a) Method A

(b) Proposal

Fig. 3. Probability distributions of the preliminary experimental results

from reference images. From Figs. 3, the threshold for MethodA is 0.34 which is the maximum of the matching rates whenimages are taken from different barcodes and The threshold forproposal is 0.11 since the proposal the blue and red bars areclassified into two around matching rate 0.11 as shown in Fig.3b. To evaluate the performance of these identifications, TablesIII show confusion matrices of the identification results. FromTables III and Equation 2, recall rates of the identificationswith the two methods are calculated.

Recall rate of the Method A is the following.

Recall rate =8

8 + 92× 100 = 8% (3)

Recall rate of the proposal is the following.

Recall rate =100

100 + 0× 100 = 100% (4)

The recall rate of the proposal is larger than that of theMethod A, then comparison of feature coordinates contributesto increase recall rate. The recall rate of the proposal is

100%, then the proposal enables individual identification of100 barcodes of single brand products completely. It alsoenables to identify barcodes when a matching rate is more thanabout 10% since calculated matching rates can be classifiedinto two parts around matching rate 0.11.

TABLE IIICONFUSION MATRICES OF THE PRELIMINARY EXPERIMENTAL RESULTS

Positive Negative+ 8 92− 0 9900

(a) Method A

Positive Negative+ 100 0− 0 9900

(b) Proposal

VII. EVALUATION WITH EXPERIMENT

A. Method

To evaluate quantitatively the performance of the proposalwhen processing product-level and individual identifications,an experiment are performed by implementing an experimentalsystem which performs the two identifications. Table IV showsan environement of the system. The system performs affinetransform using edge based transformation in the geometrictransform phase. The transformation is based on correspon-dences of edges of rectangles of barcode bars. The systemprocesses barcode images taken from 20 box products of twobrands. For product-level identification, patterns of black barsand white spaces have to be taken. In order to take the patternsand clear outlines of barcodes, the images are taken with acompact digital camera using its macro photographic mode.Table V shows a specification of the camera. Each barcodewas taken twice for both a stored reference and an input query.To evaluate results of matching features, numbers of matchingfeatures are converted to a matching rate as shown in Equation5.

Matching rate =nmatch

nquery(5)

To describe the performance of the identification, a confusionmatrix is assigned as same as the preliminary experiment.

B. Result

The system computed 100 patterns of feature matchingbetween 10 template images and 10 input images for eachproduct. Figure 4a shows the probability distribution of match-ing rates with Product A and Figure 4b shows that withProduct B. The contents of these figures are same as thepreliminary experiment.

C. Consideration

This experiment assumes that the identification system is seta threshold of the matching rate to identify query images fromreference images. From Figs. 4, the thresholds for Product Aand Product B is 0.55 and 0.76 which are the maximums of thematching rates when images are taken from different barcodes.To evaluate the performance of these identifications, TablesVI show confusion matrices of the identification results. From

TABLE IVENVIRONMENT OF THE EXPERIMENTAL SYSTEM

OS Ubuntu 16.04 LTSProgramming Language Python 3

Libraries OpenCV 3.4.0

TABLE VSPECIFICATION OF THE CAMERA (OLYMPUS, TG-5)

camera sensor 1200 million pixel BSI-CMOSmagnification 4-fold

focus length(35mm camera equivalent)

4.5mm - 18.0mm(25mm - 100mm)

Tables VI and Equation 2, recall rates of the identificationswith the two methods are calculated.

Recall rate of Product A is the following.

Recall rate =6

6 + 4× 100 = 60% (6)

Recall rate of Product B is the following.

Recall rate =9

9 + 1× 100 = 90% (7)

The system can perform product identification completelyand individual identification of Product A with 60% recall rate.The system can perform product identification completely andindividual identification of Product B with 90% recall rate. In abest case, the proposal are experimentally evaluated to performproduct identification completely and individual identificationof commercial products with about 90% recall rate.

VIII. CONCLUSION

Product-level barcodes, extensively used everyday, can beused as a mean to perform individual identification of com-mercial products. Distinctive features barcodes can be foundin the subtle but distinctive print irregularities in product-levelbarcodes. Individual identification using print irregularities inbarcodes, referred to as Barcode Fingerprinting, is realized byextracting features of color gradients and matching them inlocations and gradient vectors between images. Speed-Up Ro-bust Feature (SURF) algorithm can be used to extract featurestaking an advantage its characteristic which it extract samefeatures repeatedly thanks of its scale- and rotation- invariance.The proposal with microscope cameras can achieve almost100% recall rate experimentally verified, however, in this case,it is necessary to take a barcode twice – once for product-level identification and once for individual identification of

TABLE VICONFUSION MATRICES OF THE EXPERIMENTAL RESULTS

Positive Negative+ 6 4− 0 90

(a) Product A

Positive Negative+ 9 1− 0 90

(b) Product B

(a) Product A

(b) Product B

Fig. 4. Probability distributions of the experimental results

a product. Further, this experiment reveals that images aretaken from a single barcode when about the 10% featuresare same between the images. The proposal with compactdigital cameras which have macro photographic mode canarchive product-level identification completely and individualidentification with almost 90% recall rate in a best caseexperimentally verified.

I presented our research on February 2018 [4].

REFERENCES

[1] M. L. Soborski, “Unique identification information from marked fea-tures,” Aug. 3 2017, uS Patent App. 15/491,523.

[2] T. Takahashi and R. Ishiyama, “FIBAR: Fingerprint Imaging by BinaryAngular Reflection for Individual Identification of Metal Parts,” in 2014Fifth International Conference on Emerging Security Technologies, 2014,pp. 46–51.

[3] J. Wayman, A. Jain, D. Maltoni, and D. Maio, “An introduction tobiometric authentication systems,” in Biometric Systems. Springer, 2005,pp. 1–20.

[4] R. Ueno and J. Mitsugi, “Barcode fingerprinting: Unique identification ofcommercial products with their JAN/EAN/UCC barcode,” pp. 416–420,2018.