Bioresource Technology 100 (2009) 4499–4506

Contents lists available at ScienceDirect

Bioresource Technology

journal homepage: www.elsevier .com/locate /bior tech

Using parabolic mirrors for complete imaging of apple surfaces

Daniel Reese a,b, Alan M. Lefcourt b,*, Moon S. Kim b, Y. Martin Lo a

a Department of Nutrition and Food Science, University of Maryland, College Park, MD 20742, USAb Henry A. Wallace Beltsville Agricultural Research Center, USDA-ARS, Environmental Microbial and Food Safety Laboratory, Building 303, Beltsville, MD 20705, USA

a r t i c l e i n f o

Article history:Received 2 January 2008Received in revised form 10 November 2008Accepted 20 November 2008Available online 9 May 2009

Keywords:ImagingParabolic mirrorsApplesMethodMachine vision

0960-8524/$ - see front matter Published by Elsevierdoi:10.1016/j.biortech.2008.11.059

* Corresponding author. Tel.: +1 301 5048450; fax:E-mail address: alan.lefcourt@ars.usda.gov (A.M. L

a b s t r a c t

Automated imaging systems offer the potential to inspect the quality and safety of fruits consumed bythe public. One problem that has hindered adoption of automated technologies has been the inabilityto image the complete surface of an individual fruit. A particular problem is that both the stem and calyxare concave structures. The goal of this project was to examine tradeoffs for using multiple mirrors toimage the surface of apples. For testing, individual apples were suspended using two thin wires, mirrorswere placed around an apple, and movies were captured at 90 images per sec. Apples were rotated in alldimensions to examine the efficacy of different mirror configurations. It was determined that specificconfigurations of two, four, or six parabolic concave mirrors could image an entire surface. A configura-tion using two mirrors and multiple images acquired as apples roll by was also found to be viable.

Published by Elsevier Ltd.

1. Introduction

Machine vision is increasingly used for automated inspection ofagricultural commodities (Brosnan and Sun, 2004; Chen et al.,2002). For fruit such as apples, commercial systems are availablethat allow sorting by size, shape and color. However, inspectionfor damaged, diseased, or contaminated fruit has been hinderedby the inability to appropriately orient fruit for imaging and by lackof a method for imaging 100% of the surface of individual fruit. Re-search results suggest that it is feasible to use machine vision sys-tems to inspect fruit for quality related problems (Bennedsen andPeterson, 2005; Brosnan and Sun, 2004; Kleynen et al., 2005; Mehlet al., 2004; Throop et al., 2005) and for fecal contamination(Kim et al., 2002; Lefcourt et al., 2003, 2005). Currently, sortingof fruits for surface defects is mainly done by manual inspection(Bennedsen and Peterson, 2005) and no commercial system isavailable for detecting fecal contamination. This study examinesthe use of mirrors to enable 100% of an apple’s surface to be imagedusing a single camera.

The concave nature of the stem and calyx regions presents amajor problem for image acquisition and analyses. To image theentire surface requires sufficient perspectives to guarantee thatcameras or mirrors can ‘‘see” inside these regions. One solutionto this problem is to orient the apple prior to imaging so that thelocations of the stem and calyx regions are known during imaging.If apple orientation can be controlled, mirrors (or cameras) could

Ltd.

+1 301 504 9466.efcourt).

be situated opposite the expected locations of these regions. Itwas recently discovered that apples could be oriented along thestem/calyx axis using their inertial properties (Narayanan et al.,2008b; Tasch, 2006). When apples were rolled down woodentracks consisting of two parallel rails, they generally moved to anorientation where the stem/calyx axis was parallel to the planeof the track and perpendicular to the direction of travel (Narayananet al., 2008a). The initial concept for imaging the oriented appleswas to transition the apples from the two parallel rails to two par-allel wires. Wires were used to minimize the area of the apple ob-scured by support structures in acquired images. Preliminary testsusing two parabolic concave mirrors were encouraging (Reeseet al., 2007; Fig. 1). In reflected images, only the areas near the cir-cumference that was parallel to the wires, i.e., the edges of the re-flected images, were poorly visualized. One goal of this study wasto determine if mirror positioning could be altered to allow bettervisualization of the areas near the circumference.

It is not clear if it will be possible to orient all apples based-ontheir inertial properties. In tests of the orientation process, the suc-cess rate for orientation of apples with a particular initial orienta-tion was unacceptably low at about 50% (Narayanan et al.,2008b). Further tests are needed to determine if problems due tothis initial condition can be circumvented. If apples are not appro-priately oriented for imaging, either by choice or because orienta-tion technology proves inadequate, the orientation of applesduring imaging will be random. Another goal of this study was todetermine if the entire surface of a randomly oriented apple, includ-ing the inside areas of the stem and calyx, could be captured using asingle camera and multiple mirrors. Any solution would also applyto the imaging of other commodities with a structure similar to an

Fig. 2. Schematic representations of using two, four, or six parabolic concavemirrors for image acquisition that show an apple supported by two wiressurrounded by mirrors with reflected (R) images of the apple. For the two mirrorcase, the effects of the apple rolling on the support wires is depicted to show thatthe visible portion of the apple changes over time.

Fig. 1. Color image acquired using two flat (a) or concave parabolic (b) mirrors. Dueto apple orientation, mirrors are able to fully see inside the stem and calyx regions.

4500 D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506

apple, i.e., any ‘‘round” object that includes a concave structure aspart of the surface.

2. Methods

The test apparatus consisted of two parallel thin wires (4-gaugemusic wire) used to support apples, a monochrome video camera(EC650, Prosilica; 640 � 480 pixels, non-interlaced, 90 frames persec) mounted 140 cm above the wires, and a fixed-focal length lens(Schneider Xenoplan, 17 mm, f/1.4). Wires were four cm apart andwere supported 19 cm above a steel optical bench. Halogen light-ing reflected off a curved white surface above the apple providedillumination. Software written in Visual Basic 6 allowed moviesto be acquired and displayed in real time. This test system allowedresults of changes in mirror and apple position to be examined on-the-fly. As needed, small pieces of tape were added to the wires tohold an apple in a selected position.

Tests were conducted using rectangular flat and parabolic mir-rors with 3x magnification (20.2 � 12.7 cm), and round parabolicmirrors with 5x magnification (13.0 cm dia.). Rather than con-structing complete mirror configurations, a decision was made touse sets of images and one or two mirrors to capture the informa-tion that would be available in a single image using a particularmirror configuration. For example, for radial placements of mirrorsaround an apple, sets of images were acquired using two opposingmirrors with the apple rotated between individual image acquisi-tions at the angle between mirrors. Images of reflected surfaceswere analyzed in terms of two potential imaging conditions: acqui-sition of a single image and acquisition of multiple images as theapple rolled through the imaging area on the two wires.

The ability to monitor effects of changes in mirror and applepositioning on-the-fly was used to identify ranges of values formore detailed study. First, the angle and location of test mirrorswere modified so that direct and reflected images were separatedby a minimal distance in acquired images. For each mirror inclina-tion, effects on distortion were noted. Using results of these empir-ical tests, sets of test conditions for more detailed study wereselected. For the detailed tests, the inclination of the mirrors andthe vertical and lateral angles for apple locations were set using aprotractor.

Data were analyzed by looking at sequences of images whereone parameter was incremented. For presentation, the optimaland worst-case scenarios were selected. Worst-case was definedin terms of the difficulty of seeing inside the stem or calyx, or somearea of the surface, in the set of images that equated to a single im-age using a particular mirror configuration. To allow visual assess-ment of the degree of a problem, the image with the problem areawas depicted along with the surrounding incremented images. Tofacilitate localization of problem areas, lines were drawn on applesand round stickers were placed at critical locations. Images werecropped for display. For images with two mirrors the native pixeldimensions were 560 by 180. Cropped images were transformedto 600 dpi for publication.

3. Results and discussion

3.1. Mirror configurations

Four conceptual mirror configurations are shown in Fig. 2. Onlymirror configurations with an even number of mirrors were con-sidered for two reasons; apples are bilaterally symmetric and thesupport wires effectively divide the imaging area into two bilater-ally symmetric fields. Data are presented only for parabolic mirrorsas preliminary tests demonstrated their superiority to flat mirrors.In comparison with flat mirrors, concave parabolic mirrors magni-fying the size of apples in images and reduce contrast resultingfrom reflection of light from the curved surfaces of apples. Theunderlying optical principles for image magnification using eitherconcave parabolic mirrors or telephoto lenses are similar. As witha telephoto lens, as magnification increases the actual area beingimaged decreases and any object that remains within the imagingarea occupies a greater percentage of the total pixels of the imag-ing device. The advantage of using mirrors is that magnificationcan be applied selectively to multiple focal areas. For example, gi-ven the same camera and mirror positioning, an apple encom-

D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506 4501

passed 12.9% of the total number of image pixels when flat mirrorswere used versus 17.0% when parabolic mirrors were used (Fig. 1).If the same level of magnification resulting from the use of para-bolic mirrors was applied to the whole imaging area using a tele-photo lens, the resulting shrinkage of the imaging field wouldeliminate the mirrors from acquired images. Parabolic mirrors im-prove image contrast due to a collimating effect.

Only incidental results for the angled four mirror configurationare presented. For the angled four mirror configuration, theworst-case positioning of the stem/calyx axis is laterally 0� or 90�relative to the support wires. Most, if not all, apples are expectedto be oriented with the stem/calyx axis perpendicular to the sup-port wires. It makes little sense to study a configuration wherethe most common orientation is also a worst-case orientation. Inaddition, results will show that a 30� offset of the stem/calyx axisfrom the axis of paired-opposing mirrors can create problems forimaging inside the stem or calyx. The 30� angle is half the angle sep-arating mirrors in the six mirror configuration. A 45� offset for theangled four mirror configuration would create even more of aproblem.

3.2. Additional considerations

Detection of nanogram quantities of feces on apples is more dif-ficult when the feces are at the edge of an apple surface in an image(Lefcourt et al., 2003). Thus, the optimal imaging configurationwould produce sufficient imaging perspectives so that 100% ofthe surface could be analyzed without having to look near edges.Second, for safety inspection the goal is detection and not quanti-fication. This goal allows consideration of configurations that causeshape distortions without requiring that the distortions be pre-cisely mapped. Similarly, the existence of redundant informationthat might result from replicate sampling of some areas of the sur-face is not a problem. The only concern is that 100% of the surfaceis well represented.

Fig. 3. The 0� images depict typical images that would be acquired using the two mirroraspects that would be acquired using the six mirror configuration. The mirror under wirefour mirror configuration.

3.3. Oriented apple imaging

Some perspectives that might be available for imaging an ori-ented apple are depicted in Fig. 3. The top two images show theinformation available using the two mirror configuration. The ap-ple has large round stickers placed on the sides of the apples inthe top and the bottom of theses images. These stickers are barelyevident regardless of mirror inclination. Thus, while inclining themirrors allows visualization of the bottom center of the apple,the areas with the stickers still do not appear in the acquiredimages. The 60� images depict the additional information thatwould be available if a six mirror configuration was used. Theround stickers are clearly visible. While one edge the sticker forthe 25� mirror inclination appears to be precariously near the edgeof the apple, the section of the sticker near the edge will be re-versed in the �60� image (not depicted). The mirror under wiresimage depicts the additional information that would be availableif the four mirror configuration was used. The bottom of the appleand the sticker are clearly visible. Thus, both the four and six mir-ror configurations are adequate to allow visualization of 100% of anoriented apple’s surface, minus the area obscured by the supportwires. It should be noted that imaging using multiple angles (mir-rors) reduces the already small interference of the support wires asonly the points actually or nearly touching the apple are notimaged.

3.4. Randomly oriented apple imaging

A range of possible scenarios for imaging using the six mirrorconfiguration are depicted in Figs. 4 and 5. One of the two most dif-ficult imaging situations arises when the apple is or is approachingupright. Under these conditions, the inside of the bottom stem orcalyx is not completely rendered for the 40� mirror inclination.The other problem situation occurs when the apple is laterally ro-tated 30� and horizontally rotated 45�. In this case the problem is

configuration for an oriented apple. The 60� images depict one of the two additionals image depicts one of the two additional aspects that would be acquired using the

Fig. 4. Sequence of images using two mirrors at a higher (40�) and lower (25�) angle of inclination. The stem/calyx axis is perpendicular to the parallel support wires and theapple is rotated vertically from 0� to 90� at 15� increments. Note that with the 25� inclination the inside of the calyx is visible when the apple is fully vertical, but the stem isnot visible with the 40� inclination.

4502 D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506

most evident with the 25� mirror inclination. Overall, the occur-rence rate of imaging problems with randomly oriented applesshould be low for both mirror inclinations and lowest for the 25�inclination.

The range of possible scenarios for imaging using the four mir-ror configuration is depicted in Figs. 6 and 7. There appears to be a

small loss of information when the apple is laterally rotated 45� atthe lower horizontal rotations. There also may be some loss ofinformation when the apple is laterally rotated 90� and horizon-tally rotated 45�. In general, the interior of the stem and calyx re-gions is less well represented than is the case for the six mirrorconfiguration. Still, the lost information represents only a small

Fig. 5. Sequence of images using two mirrors at a higher (40�) and lower (25�) angle of inclination. The stem/calyx axis is laterally rotated 30� from the parallel support wiresand the apple is rotated vertically from 0� to 90� at 15� increments. Note that for the 25� inclination a small amount of data about the inside of the calyx is lost at 45�.

D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506 4503

portion of potential information in the stem and calyx areas. Thereal problem with the four mirror configuration is that two ofthe mirrors are under the support wires. These mirrors could be oc-cluded if apples were close together as they rolled through theimaging area. The speed and acceleration of apples as they rolldown the orientation test tracks varies (Narayanan et al., 2008b).Thus, the need to guarantee a minimum spacing between apples

during imaging could reduce the potential maximum apple pro-cessing rate.

3.5. Imaging rolling apples

There is an alternative to the four mirror configuration. Insteadof using a single acquired image for detection, images could be ac-

4504 D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506

quired as the apple rolls through the imaging field as shown inFig. 2. This solution would not necessarily require additional

Fig. 6. Sequence of images using two mirrors at a higher (40�) angle of inclination along wworst-case scenario where the stem/calyx axis is laterally rotated 45� from the parallel

images to be acquired, it would just be necessary to discern thelocation of an individual apple in a sequence of images given that

ith perpendicular images from a mirror under the support wires. This represents thesupport wires. The apple is rotated vertically from 0� to 90� at 15� increments.

D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506 4505

the images might contain multiple apples. Imagining rate could befixed or a function of the location of apples on the track. However,

Fig. 7. Sequence of images using two mirrors at a higher (40�) angle of inclination alongaxis is laterally rotated 90� from the parallel support wires. The apple is rotated vertica

multiple images per apple would be needed due to variable applerotation rates and the randomness of presentation of the stem and

with perpendicular images from a mirror under the support wires. The stem/calyxlly from 0� to 90� at 15� increments.

Fig. 8. Information available uses three sequential images as the apple rollsthrough the imaging area. The apple is in an oriented position.

4506 D. Reese et al. / Bioresource Technology 100 (2009) 4499–4506

calyx regions. One benefit of this imaging scheme is the elimina-tion of the need for support wires. Apples could be imaged fromabove as the apples rolled down the orientation track. Fig. 8 showsthe information that might be acquired for an oriented apple usingthis imaging scheme.

3.6. Existing imaging solutions

Currently, mirrors are not commonly used in commercial agri-cultural processing systems, primarily due to problems with dirtaccumulation. Literature searches and online searches of commer-cial sorting systems failed to provide any evidence of use of para-bolic concave mirrors in machine vision imaging systems. Theincreased resolution and decreased distortion at the edges ofimages acquired using this type of mirror warrant considerationof their use.

An imaging method demonstrated by Li et al. (2002) uses a cupholder with a hole in the bottom along with a camera below thecup and a camera above to image most of a fruit’s surface. The ma-jor problems with this method are that the entire fruit cannot beimaged due to blockage by the cup and the processing speed of3–4 apples per sec is lower than commercial speeds of 10 applesper sec. Furthermore, the camera below is just as likely as a mirrorto get dirty, and the use of more than one camera diminishes thedesirability of this system due to increased cost and complexity.

4. Conclusions

Acquiring images representative of 100% of the surface of applesis difficult due to the concave nature of the stem and calyx regions.

To test if mirrors could be used to image 100% of the surface of ap-ples, configurations of two, four, and six mirrors were tested. Re-sults demonstrated that single images acquired using the four orsix mirror configurations, or a single image acquired using thetwo mirror configuration along with multiple images acquired asthe apple rolled through the imaging field, could be used to imagealmost 100% of the surface of apples regardless of apple orienta-tion. However, all configurations worked best if the apples wereoriented so that the stem region faced one mirror and the calyx re-gion faced the opposing mirror. Parabolic concave mirrors werefound to be superior to flat mirrors due to their ability to magnifyimages and thereby increase the effective resolution of acquiredimages, and to reduce contrast in images resulting from reflectionof light from the curved surfaces of apples. These results suggestthat consideration for using parabolic mirrors for commercial ap-ple inspection is warranted.

References

Bennedsen, B.S., Peterson, D.L., 2005. Performance of a system for apple surfacedefect identification in near-infrared images. Biosyst. Eng. 90 (4), 419–431.

Brosnan, T., Sun, D.W., 2004. Improving quality inspection of food products bycomputer vision – a review. J. Food Eng. 61 (1), 3–16.

Chen, Y.R., Chao, K., Kim, M.S., 2002. Machine vision technology for agriculturalapplications. Comput. Electron. Agric. 33 (2/3), 173–191.

Kim, M.S., Lefcourt, A.M., Chen, Y.R., Kim, I., Chan, D.E., Chao, K., 2002. Multispectraldetection of fecal contamination on apples based on hyperspectral imagery:part II. Application of hyperspectral fluorescence imaging. Trans. ASAE 45 (6),2039–2047.

Kleynen, O., Leemans, V., Destain, M.F., 2005. Development of a multi-spectralvision system for the detection of defects on apples. J. Food Eng. 69 (1), 41–49.

Lefcourt, A.M., Kim, M.S., Chen, Y.R., 2003. Automated detection of fecalcontamination of apples by multispectral laser-induced fluorescence imaging.Appl. Opt. 42 (19), 3935–3943.

Lefcourt, A.M., Kim, M.S., Chen, Y.R., 2005. Detection of fecal contamination onapples with nanosecond-scale time-resolved imaging of laser-inducedfluorescence. Appl. Opt. 44 (7), 1160–1170.

Li, Q., Wang, M., Gu, W., 2002. Computer vision based system for apple surfacedefect detection. Comput. Electron. Agric. 36 (2–3), 215–223.

Mehl, P.M., Chen, Y.R., Kim, M.S., Chan, D.E., 2004. Development of hyperspectralimaging technique for the detection of apple surface defects andcontaminations. J. Food Eng. 61 (1), 67–81.

Narayanan, P., Lefcourt, A.M., Tasch, U., Rostamian, R., Kim, M.S., 2008a. Orientationof apples using their inertial properties. Trans. ASABE 51 (6), 2073–2081.

Narayanan, P., Lefcourt, A.M., Tasch, U., Rostamian, R., Grinblat, A., Kim, M.S., 2008b.Theoretical analysis of stability of axially symmetric rotating objects withregard to orienting apples. Trans. ASABE 54 (4), 1353–1364.

Reese, D.Y., Lefcourt, A.M., Kim, M.S., Lo, Y.M., 2007. Whole surface imagereconstruction for machine vision inspection of fruit. In SPIE MeetingPresentation. Boston, MA. Paper No. 6761-25.

Tasch, U., Narayanan, P., Lefcourt, A.M., Kim, M.S., Grinblat, A., 2006. Apparatus andMethod for Orienting Rotatable Objects. US Patent Application No.20060225582.

Throop, J.A., Aneshansley, D.J., Anger, W.C., Peterson, D.L., 2005. Quality evaluationof apples based on surface defects, development of an automated inspectionsystem. Postharvest Biol. Technol. 36 (3), 281–290.