Download - In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones
![Page 1: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/1.jpg)
In-Place Vision-Based Skewed 1D Barcode Scanning &
Effective Use of Nutrition Labels on Smartphones
Vladimir KulyukinDepartment of Computer Science
Utah State University
vkedco.blogspot.com CS Graduate Research Seminar, USU, 10/30/14
![Page 2: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/2.jpg)
Outline
● Introduction● R&D Background● An Algorithm for In-Place 1D Skewed Barcode Scanning with Relaxed
Pitch, Roll, and Yaw Constraints● 1D Skewed Barcode Scanning Experiments & Results● Effective Use of Nutrition Labels on Smartphones● Current Challenges
CS Graduate Research Seminar, USU, 10/30/14
![Page 3: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/3.jpg)
Introduction: Camera Alignment
● A common requirement for many mobile barcode scanners, both commercial and open source, is camera alignment: the camera must be aligned with a target barcode to obtain at least one complete scanline
● This requirement presents difficulties for visually impaired smartphone users or for users who do not have adequate control of their hands
● Another challenge is lack of coupling of barcode scanning to comprehensive nutrition label databases
CS Graduate Research Seminar, USU, 10/30/14
![Page 4: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/4.jpg)
Introduction: Nutrition Labels
● Many products sold worldwide have nutrition labels (NLs) ● In the U.S., the display of nutrition information is mandated by the
Nutrition Education and Labeling Act (NLEA) of 1990● EU FLABEL is a similar legislative act in EU● Even highly motivated consumers find it difficult to locate and
comprehend nutrition information on many products● Due to their ever increasing presence, smartphones are an
excellent platform for proactive nutrition management
CS Graduate Research Seminar, USU, 10/30/14
![Page 5: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/5.jpg)
Two Great Books
CS Graduate Research Seminar, USU, 10/30/14
![Page 6: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/6.jpg)
R& D Background
CS Graduate Research Seminar, USU, 10/30/14
![Page 7: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/7.jpg)
Dominant Orientation of Gradients● In our previous work (Kulyukin et al., IPCV 2013), we
developed a vision-based algorithm for localizing skewed barcodes on smartphones (pdf)
● The algorithm localizes skewed barcodes in captured frames by computing dominant orientation of gradients (DOGs) of image segments and collecting smaller segments with similar DOGs into larger connected components (DOGs eat barcodes :-))
● The algorithm was successfully evaluated on 1,066 images
CS Graduate Research Seminar, USU, 10/30/14
![Page 8: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/8.jpg)
Dominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of Gradients
Regions with similar DOGs are combined into clusters on the basis of their geometric proximity
CS Graduate Research Seminar, USU, 10/30/14
![Page 9: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/9.jpg)
An Algorithm for In-Place 1D Barcode Scanning
with Relaxed Pitch, Roll, and Yaw Constraints
CS Graduate Research Seminar, USU, 10/30/14
![Page 10: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/10.jpg)
Pitch, Roll, & Yaw
CS Graduate Research Seminar, USU, 10/30/14
![Page 11: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/11.jpg)
Algorithm in Action
Online demo videos:- lab video 1- lab video 2- supermarket video 1- supermarket video 2
CS Graduate Research Seminar, USU, 10/30/14
![Page 12: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/12.jpg)
Algorithm's Flowchart
CS Graduate Research Seminar, USU, 10/30/14
![Page 13: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/13.jpg)
Growth of Scanlines
Scanlines are grown orthogonally to the skew line detected by the the DOG algorithm
CS Graduate Research Seminar, USU, 10/30/14
![Page 14: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/14.jpg)
Growth of Scanlines
The number of grown scanlines is a parameter currently set to 10.
CS Graduate Research Seminar, USU, 10/30/14
![Page 15: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/15.jpg)
Linux Cluster for Image Processing &
Data Storage
CS Graduate Research Seminar, USU, 10/30/14
![Page 16: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/16.jpg)
Hardware
● A Linux cluster was originally built out of four Dell computers; two nodes were added later
● Each node computer has an Intel Core i5-6503.2 GHz dual-core processor that supports 64-bit computing
● The node's processor has 3MB of cache memory● Each node has 6GB DDR3 SDRAM and 320GB of hard
disk space
CS Graduate Research Seminar, USU, 10/30/14
![Page 17: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/17.jpg)
Software
● Ubuntu 12.04 LTS was installed on each machine● JBoss was used to build and configure the cluster● JBoss AS 7.1.1 was installed on every node● Load balancing is done with the Apache mod_cluster
module● The cluster has one master and five slaves● All machines are part of a local area network
CS Graduate Research Seminar, USU, 10/30/14
![Page 18: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/18.jpg)
Software ● The system's front end was implemented as a smartphone
application that runs on Android 4.3 or higher● The system's back end was deployed on the Linux cluster● A Java servlet for image recognition was deployed on the
master node● The servlet receives images from smartphones through
HTTP POST requests and sends nutrition labels back to the smartphones
CS Graduate Research Seminar, USU, 10/30/14
![Page 19: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/19.jpg)
1D Skewed Barcode Scanning Experiments &
Results
CS Graduate Research Seminar, USU, 10/30/14
![Page 20: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/20.jpg)
Experiment Design ● The system was evaluated on a set of 506 video recordings
of common grocery products● The videos have a 1280x720 resolution, an average
duration of 15 seconds and were recorded on an Android 4.2.2 Galaxy Nexus smartphone in a supermarket
● The videos covered four product categories: bags, boxes, bottles, and cans
● The videos are publicly available here
CS Graduate Research Seminar, USU, 10/30/14
![Page 21: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/21.jpg)
Experiment Design
● There are 130 box videos, 127 bag videos, 125 bottle videos, & 124 can videos
● Frames were extracted at a rate of 1 frames per second, which resulted in 7,545 frames
● Frames were manually classified into 1950 box frames, 1905 bag frames, 1875 bottle frames, and 1860 can frames
● Manually classified frames were automatically classified as blurred or sharp with an algorithm based on 2D Haar filter (Python source is at https://github.com/VKEDCO/PYPL/blob/master/haar_blur)
CS Graduate Research Seminar, USU, 10/30/14
![Page 22: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/22.jpg)
Average Request-Response Times (ms)
16 sessions of 100 frames per session; tests were done over a Wi-Fi network with a download speed of 72.31 Mpbs and an upload speed of 24.64 Mbs
CS Graduate Research Seminar, USU, 10/30/14
![Page 23: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/23.jpg)
Impact of Blurriness
CS Graduate Research Seminar, USU, 10/30/14
![Page 24: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/24.jpg)
Average Scan Times in Product Categories
CS Graduate Research Seminar, USU, 10/30/14
![Page 25: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/25.jpg)
Results● In all four categories of products, the true negative and false positive
percentages are 0● The false negative percentages are relatively high: the algorithm
rejects frames on the slightest chance that it does not contain a barcode
● The algorithm scans barcodes of any orientation in the yaw plane, of up to 73 degrees in the pitch plane, and of up to 55 degrees in the roll plane
CS Graduate Research Seminar, USU, 10/30/14
![Page 26: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/26.jpg)
NutriGlass 1.0 on Google Play
The free front end application is available on Google Play; search “NutriGlass on Google Play” to download; make sure that OpenCV libraries are available on your Android platform; online demo is here.
CS Graduate Research Seminar, USU, 10/30/14
![Page 27: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/27.jpg)
Effective Use of Nutrition Labels on
Smartphones
CS Graduate Research Seminar, USU, 10/30/14
![Page 28: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/28.jpg)
Barriers to NL Comprehension
● Recent investigations of NL use by consumers have used digital cameras to track consumers' eye movements to understand how consumers locate and understand NLs
● Four key barriers have been identified in the literature: 1) label's location on the package; 2) presentation of information within the label; 3) label's surface size; 4) surrounding visual clutter
CS Graduate Research Seminar, USU, 10/30/14
![Page 29: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/29.jpg)
Confusing NL Examples
CS Graduate Research Seminar, USU, 10/30/14
![Page 30: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/30.jpg)
Confusing NL Examples
CS Graduate Research Seminar, USU, 10/30/14
![Page 31: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/31.jpg)
Recommendations from NL Use Literature
1) Central positions of NLs2) Nutrients sorted by health relevance3) Explanation of nutrients4) Reduced visual clutter around NLs 5) Increased visual salience through contrast and orientation6) Increased surface size of NLs
CS Graduate Research Seminar, USU, 10/30/14
![Page 32: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/32.jpg)
Implementation Details
● The front end application sends captured frames to the back end wirelessly
● The back end cluster does image processing (1D skewed UPC and EAN barcode scanning) and, if the scanning is successful, the scanned barcode is used to retrieve a corresponding NL from a database of 230,000 NLs
● The retrieved NL is displayed on the smartphone
![Page 33: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/33.jpg)
Sample Screenshots
CS Graduate Research Seminar, USU, 10/30/14
![Page 34: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/34.jpg)
NutriMenuNXT
NutriMenuNXT is an Android application that allows the user to get nutritional information on menu items from 296 restaurant chains; search “NutriMenuNXT on Google Play” to download; online demo is here
CS Graduate Research Seminar, USU, 10/30/14
![Page 35: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/35.jpg)
Back to Four Targeted Recommendations ● NutriGlass & NutriMenu satisfy four out of six recommendations
found in the literature on effective use of nutrition labels:
1) Central positions of NLs: NLs are displayed centrally on the smarpthone's screen3) Explanation of nutrients: Wiki links4) Reduced visual clutter around NLs: no visual clutter around the label 6) Increased surface size of NLs: NLs are displayed on the entire touchscreen
CS Graduate Research Seminar, USU, 10/30/14
![Page 36: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/36.jpg)
Current Challenges
CS Graduate Research Seminar, USU, 10/30/14
![Page 37: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/37.jpg)
Image Blur Detection● Blurred images, as experiments show, have a
negative impact on skewed barcode detection● A current challenge is to detect and filter blurred
images before sending them to the cloud● If image blur can be detected on the smartphone,
data throughput and recognition accuracy will likely improve
CS Graduate Research Seminar, USU, 10/30/14
![Page 38: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/38.jpg)
Blurred vs. Sharp Images
CS Graduate Research Seminar, USU, 10/30/14
This image should be thrown away This image should be sent to the cloud for processing
![Page 39: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/39.jpg)
Image Blur Detection● We can estimate a degree of change in various
regions of the image (this can be done with wavelets)● Hypothesis: sharp images will have clusters of
regions with pronounced changes; blurred images will either have no such regions or small clusters
● Initial experiments indicate that the hypothesis may be validated
CS Graduate Research Seminar, USU, 10/30/14
![Page 40: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/40.jpg)
High-Change Clusters in Blurred & Sharp Images
CS Graduate Research Seminar, USU, 10/30/14
Blurred Image: no high-change clusters Sharp Image: pronounced high-change cluster in the middle
![Page 41: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/41.jpg)
Nutrition Information Extraction
Line Segmentor
Table Segmentor
TTS Engine
Image
Text
Table Lines
Open CVOCR
Engine
CS Graduate Research Seminar, USU, 10/30/14
![Page 42: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/42.jpg)
Skewed Nutrition Labels & Texts● We have solved the problem of localizing aligned and slightly skewed
nutrition labels:– Kulyukin, V. and Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed
Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
– Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.
● Our current challenge is to detect highly skewed nutrition labels and to estimate the amount (in degrees) of text skew: this will allow us to detect rows and use OCR to extract information
CS Graduate Research Seminar, USU, 10/30/14
![Page 43: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/43.jpg)
References● V. Kulyukin & T. Zaman. Vision-Based Localization and Scanning of 1D UPC and EAN Barcodes with
Relaxed Pitch, Roll, and Yaw Camera A lignment Constraints. International Journal of Image Processing (IJIP), V olume (8) : Issue (5) : 2014, pp. 355-383.
● Kulyukin, V. & Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
● Kulyukin, V. & Zaman, T. An Algorithm for In-Place Vision-Based Skewed 1D Barcode Scanning in the Cloud. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 36-42, July 21-24, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
● Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.
● Kulyukin, V., Zaman, T., and Andhavarapu, S. Effective Use of Nutrition Labels on Smarphones. In Proceedings of the 15th International Conference on Internet Computing and Big Data (ICOMP 2014), pp. 93 - 99, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-227-1.
CS Graduate Research Seminar, USU, 10/30/14
![Page 44: In-Place Vision-Based Skewed 1D Barcode Scanning & Effective Use of Nutrition Labels on Smartphones](https://reader033.vdocuments.net/reader033/viewer/2022042700/55871d98d8b42a9b058b4726/html5/thumbnails/44.jpg)
Thank you!
CS Graduate Research Seminar, USU, 10/30/14