in-place vision-based skewed 1d barcode scanning & effective use of nutrition labels on...
TRANSCRIPT
In-Place Vision-Based Skewed 1D Barcode Scanning &
Effective Use of Nutrition Labels on Smartphones
Vladimir KulyukinDepartment of Computer Science
Utah State University
vkedco.blogspot.com CS Graduate Research Seminar, USU, 10/30/14
Outline
● Introduction● R&D Background● An Algorithm for In-Place 1D Skewed Barcode Scanning with Relaxed
Pitch, Roll, and Yaw Constraints● 1D Skewed Barcode Scanning Experiments & Results● Effective Use of Nutrition Labels on Smartphones● Current Challenges
CS Graduate Research Seminar, USU, 10/30/14
Introduction: Camera Alignment
● A common requirement for many mobile barcode scanners, both commercial and open source, is camera alignment: the camera must be aligned with a target barcode to obtain at least one complete scanline
● This requirement presents difficulties for visually impaired smartphone users or for users who do not have adequate control of their hands
● Another challenge is lack of coupling of barcode scanning to comprehensive nutrition label databases
CS Graduate Research Seminar, USU, 10/30/14
Introduction: Nutrition Labels
● Many products sold worldwide have nutrition labels (NLs) ● In the U.S., the display of nutrition information is mandated by the
Nutrition Education and Labeling Act (NLEA) of 1990● EU FLABEL is a similar legislative act in EU● Even highly motivated consumers find it difficult to locate and
comprehend nutrition information on many products● Due to their ever increasing presence, smartphones are an
excellent platform for proactive nutrition management
CS Graduate Research Seminar, USU, 10/30/14
Two Great Books
CS Graduate Research Seminar, USU, 10/30/14
R& D Background
CS Graduate Research Seminar, USU, 10/30/14
Dominant Orientation of Gradients● In our previous work (Kulyukin et al., IPCV 2013), we
developed a vision-based algorithm for localizing skewed barcodes on smartphones (pdf)
● The algorithm localizes skewed barcodes in captured frames by computing dominant orientation of gradients (DOGs) of image segments and collecting smaller segments with similar DOGs into larger connected components (DOGs eat barcodes :-))
● The algorithm was successfully evaluated on 1,066 images
CS Graduate Research Seminar, USU, 10/30/14
Dominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of Gradients
Regions with similar DOGs are combined into clusters on the basis of their geometric proximity
CS Graduate Research Seminar, USU, 10/30/14
An Algorithm for In-Place 1D Barcode Scanning
with Relaxed Pitch, Roll, and Yaw Constraints
CS Graduate Research Seminar, USU, 10/30/14
Pitch, Roll, & Yaw
CS Graduate Research Seminar, USU, 10/30/14
Algorithm in Action
Online demo videos:- lab video 1- lab video 2- supermarket video 1- supermarket video 2
CS Graduate Research Seminar, USU, 10/30/14
Algorithm's Flowchart
CS Graduate Research Seminar, USU, 10/30/14
Growth of Scanlines
Scanlines are grown orthogonally to the skew line detected by the the DOG algorithm
CS Graduate Research Seminar, USU, 10/30/14
Growth of Scanlines
The number of grown scanlines is a parameter currently set to 10.
CS Graduate Research Seminar, USU, 10/30/14
Linux Cluster for Image Processing &
Data Storage
CS Graduate Research Seminar, USU, 10/30/14
Hardware
● A Linux cluster was originally built out of four Dell computers; two nodes were added later
● Each node computer has an Intel Core i5-6503.2 GHz dual-core processor that supports 64-bit computing
● The node's processor has 3MB of cache memory● Each node has 6GB DDR3 SDRAM and 320GB of hard
disk space
CS Graduate Research Seminar, USU, 10/30/14
Software
● Ubuntu 12.04 LTS was installed on each machine● JBoss was used to build and configure the cluster● JBoss AS 7.1.1 was installed on every node● Load balancing is done with the Apache mod_cluster
module● The cluster has one master and five slaves● All machines are part of a local area network
CS Graduate Research Seminar, USU, 10/30/14
Software ● The system's front end was implemented as a smartphone
application that runs on Android 4.3 or higher● The system's back end was deployed on the Linux cluster● A Java servlet for image recognition was deployed on the
master node● The servlet receives images from smartphones through
HTTP POST requests and sends nutrition labels back to the smartphones
CS Graduate Research Seminar, USU, 10/30/14
1D Skewed Barcode Scanning Experiments &
Results
CS Graduate Research Seminar, USU, 10/30/14
Experiment Design ● The system was evaluated on a set of 506 video recordings
of common grocery products● The videos have a 1280x720 resolution, an average
duration of 15 seconds and were recorded on an Android 4.2.2 Galaxy Nexus smartphone in a supermarket
● The videos covered four product categories: bags, boxes, bottles, and cans
● The videos are publicly available here
CS Graduate Research Seminar, USU, 10/30/14
Experiment Design
● There are 130 box videos, 127 bag videos, 125 bottle videos, & 124 can videos
● Frames were extracted at a rate of 1 frames per second, which resulted in 7,545 frames
● Frames were manually classified into 1950 box frames, 1905 bag frames, 1875 bottle frames, and 1860 can frames
● Manually classified frames were automatically classified as blurred or sharp with an algorithm based on 2D Haar filter (Python source is at https://github.com/VKEDCO/PYPL/blob/master/haar_blur)
CS Graduate Research Seminar, USU, 10/30/14
Average Request-Response Times (ms)
16 sessions of 100 frames per session; tests were done over a Wi-Fi network with a download speed of 72.31 Mpbs and an upload speed of 24.64 Mbs
CS Graduate Research Seminar, USU, 10/30/14
Impact of Blurriness
CS Graduate Research Seminar, USU, 10/30/14
Average Scan Times in Product Categories
CS Graduate Research Seminar, USU, 10/30/14
Results● In all four categories of products, the true negative and false positive
percentages are 0● The false negative percentages are relatively high: the algorithm
rejects frames on the slightest chance that it does not contain a barcode
● The algorithm scans barcodes of any orientation in the yaw plane, of up to 73 degrees in the pitch plane, and of up to 55 degrees in the roll plane
CS Graduate Research Seminar, USU, 10/30/14
NutriGlass 1.0 on Google Play
The free front end application is available on Google Play; search “NutriGlass on Google Play” to download; make sure that OpenCV libraries are available on your Android platform; online demo is here.
CS Graduate Research Seminar, USU, 10/30/14
Effective Use of Nutrition Labels on
Smartphones
CS Graduate Research Seminar, USU, 10/30/14
Barriers to NL Comprehension
● Recent investigations of NL use by consumers have used digital cameras to track consumers' eye movements to understand how consumers locate and understand NLs
● Four key barriers have been identified in the literature: 1) label's location on the package; 2) presentation of information within the label; 3) label's surface size; 4) surrounding visual clutter
CS Graduate Research Seminar, USU, 10/30/14
Confusing NL Examples
CS Graduate Research Seminar, USU, 10/30/14
Confusing NL Examples
CS Graduate Research Seminar, USU, 10/30/14
Recommendations from NL Use Literature
1) Central positions of NLs2) Nutrients sorted by health relevance3) Explanation of nutrients4) Reduced visual clutter around NLs 5) Increased visual salience through contrast and orientation6) Increased surface size of NLs
CS Graduate Research Seminar, USU, 10/30/14
Implementation Details
● The front end application sends captured frames to the back end wirelessly
● The back end cluster does image processing (1D skewed UPC and EAN barcode scanning) and, if the scanning is successful, the scanned barcode is used to retrieve a corresponding NL from a database of 230,000 NLs
● The retrieved NL is displayed on the smartphone
Sample Screenshots
CS Graduate Research Seminar, USU, 10/30/14
NutriMenuNXT
NutriMenuNXT is an Android application that allows the user to get nutritional information on menu items from 296 restaurant chains; search “NutriMenuNXT on Google Play” to download; online demo is here
CS Graduate Research Seminar, USU, 10/30/14
Back to Four Targeted Recommendations ● NutriGlass & NutriMenu satisfy four out of six recommendations
found in the literature on effective use of nutrition labels:
1) Central positions of NLs: NLs are displayed centrally on the smarpthone's screen3) Explanation of nutrients: Wiki links4) Reduced visual clutter around NLs: no visual clutter around the label 6) Increased surface size of NLs: NLs are displayed on the entire touchscreen
CS Graduate Research Seminar, USU, 10/30/14
Current Challenges
CS Graduate Research Seminar, USU, 10/30/14
Image Blur Detection● Blurred images, as experiments show, have a
negative impact on skewed barcode detection● A current challenge is to detect and filter blurred
images before sending them to the cloud● If image blur can be detected on the smartphone,
data throughput and recognition accuracy will likely improve
CS Graduate Research Seminar, USU, 10/30/14
Blurred vs. Sharp Images
CS Graduate Research Seminar, USU, 10/30/14
This image should be thrown away This image should be sent to the cloud for processing
Image Blur Detection● We can estimate a degree of change in various
regions of the image (this can be done with wavelets)● Hypothesis: sharp images will have clusters of
regions with pronounced changes; blurred images will either have no such regions or small clusters
● Initial experiments indicate that the hypothesis may be validated
CS Graduate Research Seminar, USU, 10/30/14
High-Change Clusters in Blurred & Sharp Images
CS Graduate Research Seminar, USU, 10/30/14
Blurred Image: no high-change clusters Sharp Image: pronounced high-change cluster in the middle
Nutrition Information Extraction
Line Segmentor
Table Segmentor
TTS Engine
Image
Text
Table Lines
Open CVOCR
Engine
CS Graduate Research Seminar, USU, 10/30/14
Skewed Nutrition Labels & Texts● We have solved the problem of localizing aligned and slightly skewed
nutrition labels:– Kulyukin, V. and Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed
Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
– Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.
● Our current challenge is to detect highly skewed nutrition labels and to estimate the amount (in degrees) of text skew: this will allow us to detect rows and use OCR to extract information
CS Graduate Research Seminar, USU, 10/30/14
References● V. Kulyukin & T. Zaman. Vision-Based Localization and Scanning of 1D UPC and EAN Barcodes with
Relaxed Pitch, Roll, and Yaw Camera A lignment Constraints. International Journal of Image Processing (IJIP), V olume (8) : Issue (5) : 2014, pp. 355-383.
● Kulyukin, V. & Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
● Kulyukin, V. & Zaman, T. An Algorithm for In-Place Vision-Based Skewed 1D Barcode Scanning in the Cloud. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 36-42, July 21-24, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.
● Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.
● Kulyukin, V., Zaman, T., and Andhavarapu, S. Effective Use of Nutrition Labels on Smarphones. In Proceedings of the 15th International Conference on Internet Computing and Big Data (ICOMP 2014), pp. 93 - 99, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-227-1.
CS Graduate Research Seminar, USU, 10/30/14
Thank you!
CS Graduate Research Seminar, USU, 10/30/14