first steps toward an electronic field guide for plants

INTRODUCTIONIn the past century we have made outstanding

progress in many areas of biology such as discoveringthe structure of DNA elucidating the processes of micro-and macroevolution and making the simple but criticalcalculation of the magnitude of species diversity on theplanet In this century we expect to witness an explosionof discoveries revolutionizing the biological sciencesand in particular the relation of human society to theenvironment However most scientists agree that globalenvironments face a tremendous threat as human popula-tions expand and natural resources are consumed As nat-ural habitats rapidly disappear the present century maybe our last opportunity to understand fully the extent ofour planetrsquos biological complexity

This urgency demands a marriage of biology andadvanced scientific tools new technologies are essentialto our progress in understanding global biocomplexityAs field taxonomists conduct their expeditions to theremaining natural habitats on the Earth they must haveaccess to and methods for searching through the vast

store of biodiversity information contained in ourlibraries museums and botanical gardens (Bisby 2000Edwards amp al 2000) New techniques being developedin areas of computer science such as vision graphicsmodeling image processing and user interface designcan help make this possible

One can envision that 21st century naturalists will beequipped with palm-top and wearable computers globalpositioning system (GPS) receivers and web-basedsatellite communication (Wilson 2000 Kress 2002Kress amp Krupnick 2005) These plant explorers willcomb the remaining unexplored habitats of the earthidentifying and recording the characters and habitats ofspecies not yet known to science Through remote wire-less communication the field botanist will be able toimmediately compare hisher newly collected plants withtype specimens and reference collections archived anddigitized in museums thousands of miles away Theinformation on these species gathered by botanists willbe sent with the speed of the Internet to their colleaguesaround the world This vision of discovering and describ-ing the natural world is already becoming a reality

597

Agarwal amp al bull Electronic field guide for plants55 (3) bull August 2006 597ndash610

First steps toward an electronic field guide for plants

Gaurav Agarwal1 Peter Belhumeur2 Steven Feiner2 David Jacobs1 W John Kress3 RaviRamamoorthi2 Norman A Bourg3 Nandan Dixit2 Haibin Ling1 Dhruv Mahajan2 RustyRussell3 Sameer Shirdhonkar1 Kalyan Sunkavalli2 amp Sean White2

1 University of Maryland Institute for Advanced Computer Studies AV Williams Building University of Mary-land College Park Maryland 20742 USA djacobsumiacsumdedu (author for correspondence)

2 Department of Computer Science Columbia University 450 Computer Science Building 1214 AmsterdamAvenue New York 10027 USA

3 Department of Botany MRC-166 United States National Herbarium National Museum of Natural HistorySmithsonian Institution P O Box 37012 Washington DC 20013-7012 USA

We describe an ongoing project to digitize information about plant specimens and make it available to botanistsin the field This first requires digital images and models and then effective retrieval and mobile computingmechanisms for accessing this information We have almost completed a digital archive of the collection oftype specimens at the Smithsonian Institution Department of Botany Using these and additional images wehave also constructed prototype electronic field guides for the flora of Plummers Island Our guides use a novelcomputer vision algorithm to compute leaf similarity This algorithm is integrated into image browsers thatassist a user in navigating a large collection of images to identify the species of a new specimen For exampleour systems allow a user to photograph a leaf and use this image to retrieve a set of leaves with similar shapesWe measured the effectiveness of one of these systems with recognition experiments on a large dataset ofimages and with user studies of the complete retrieval system In addition we describe future directions foracquiring models of more complex 3D specimens and for using new methods in wearable computing to inter-act with data in the 3D environment in which it is acquired

KEYWORDS Augmented reality computer vision content-based image retrieval electronic field guide inner-distance mobile computing shape matching species identification type specimens recognition wearablecomputing

through the partnership of natural history biologistscomputer scientists nanotechnologists and bioinfor-maticists (Wilson 2003 Kress 2004)

Accelerating the collection and cataloguing of newspecimens in the field must be a priority Augmentingthis task is critical for the future documentation of biodi-versity particularly as the race narrows between speciesdiscovery and species lost due to habitat destructionNew technologies are now being developed to greatlyfacilitate the coupling of field work in remote locationswith ready access to and utilization of data about plantsthat already exist as databases in biodiversity institutionsSpecifically a taxonomist who is on a field expeditionshould be able to easily access via wireless communica-tion through a laptop computer or high-end PDA criticalcomparative information on plant species that wouldallow himher to (1) quickly identify the plant in questionthrough electronic keys andor character recognition rou-tines (2) determine if the plant is new to science (3)ascertain what information if any currently exists aboutthis taxon (eg descriptions distributions photographsherbarium and spirit-fixed specimens living materialDNA tissue samples and sequences etc) (4) determinewhat additional data should be recorded (eg colors tex-tures measurements etc) and (5) instantaneously queryand provide information to international taxonomic spe-cialists about the plant Providing these data directly andeffectively to field taxonomists and collectors wouldgreatly accelerate the inventory of plants throughout theworld and greatly facilitate their protection and conser-vation (Kress 2004)

This technological vision for the future of taxonomyshould not trivialize the magnitude of the task before usIt is estimated that as many as 10 million species ofplants animals and microorganisms currently inhabit theplanet Earth but we have so far identified and describedless than 18 million of them (Wilson 2000) Scientistshave labored for nearly three centuries on systematicefforts to find identify name and make voucher speci-mens of new species This process of identification andnaming requires extremely specialized knowledge Atpresent over three billion preserved specimens arehoused in natural history museums herbaria and botan-ic gardens with a tremendous amount of informationcontained within each specimen Currently taxonomistson exploratory expeditions make new collections ofunknown plants and animals that they then bring back totheir home institutions to study and describe Usuallymany years of tedious work go by during which time sci-entists make comparisons to previous literature recordsand type specimens before the taxonomic status of eachnew taxon is determined

In the plant world estimates of the number of spe-cies currently present on Earth range from 220000 to o-

ver 420000 (Govaerts 2001 Scotland amp Wortley 2003)By extrapolation from what we have already describedand what we estimate to be present it is possible that atleast 10 percent of all vascular plants are still to be dis-covered and described (J Kress amp E Farr unpubl) Themajority of what is known about plant diversity existsprimarily as written descriptions that refer back to phys-ical specimens housed in herbaria Access to these spec-imens is often inefficient and requires that an inquiringscientist knows what heshe is looking for beforehandCurrently if a botanist in the field collects a specimenand wants to determine if it is a new species the botanistmust examine physical samples from the type specimencollections at a significant number of herbaria Althoughthis was the standard mode of operation for decadessuch arrangement is problematic for a number of reasonsFirst type specimens are scattered in hundreds of her-baria around the world Second the process of request-ing approving shipping receiving and returning typespecimens is extremely time-consuming both for theinquiring scientists and for the host herbarium It canoften take many months to complete a loan during whichtime other botanists will not have access to the loanedspecimens Third type specimens are often incompleteand even a thorough examination of the relevant typesmay not be sufficient to determine the status of a newspecimen gathered in the field New more efficientmethods for accessing these specimens and better sys-tems for plant identification are needed

Electronic keys and field guides two modern solu-tions for identifying plants have been available sincecomputers began processing data on morphological char-acteristics (Pankhurst 1991 Edwards amp Morse 1995Stevenson amp al 2003) In the simplest case of electron-ic field guides standard word-based field keys usingdescriptive couplets have been enhanced with colorimages and turned into electronic files to make identifi-cation of known taxa easier and faster More sophisticat-ed versions include electronic keys created through char-acter databases (eg Delta delta-intkeycom Lucidwwwlucidcentralorg) Some of these electronic fieldguides are now available on-line or for downloading ontoPDAs (eg Heidorn 2001 OpenKey httpwww3isrluiucedu~pheidorncgi-binschoolfieldguidecgi) whileothers are being developed as active websites that cancontinually be revised and updated (eg Flora of theHawaiian Islands httpravenelsiedubotanypacificislandbiodiversityhawaiianfloraindexhtm)

Somewhat further afield previous work has alsoaddressed the use of GPS-equipped PDAs to record in-field observations of animal tracks and behavior (egPascoe 1998 CyberTracker Conservation 2005) butwithout camera input or recognition (except insofar asthe skilled user is able to recognize on her own an animal

Agarwal amp al bull Electronic field guide for plants 55 (3) bull August 2006 597ndash610

598

and its behavior) PDAs with cameras have also beenused by several research groups to recognize and trans-late Chinese signs into English with the software eitherrunning entirely on the PDA (Zhang amp al 2002) or on aremote mainframe (Haritaoglu 2001) These sign-trans-lation projects have also explored how user interactioncan assist in the segmentation problem by selecting theareas in the image in which there are signs

We envision a new type of electronic field guide thatwill be much more than a device for wireless access totaxonomic literature It should allow for visual and tex-tual search through a digital collection of the worldrsquostype specimens Using this device a taxonomist in thefield will be able to digitally photograph a plant anddetermine immediately if the plant is new to science Forplants that are known the device should provide a meansto ascertain what data currently exist about this taxonand to query and provide information to taxonomic spe-cialists around the world

Our project aims to construct prototype electronicfield guides for the plant species represented in theSmithsonianrsquos collection in the United States NationalHerbarium (US) We consider three key components increating these field guides First we are constructing adigital library that is recording image and textual infor-mation about each specimen We initially concentratedour efforts on type specimens (approximately 85000specimens of vascular plants at US) because they arecritical collections and represent unambiguous speciesidentification We envision that this digital collection atthe Smithsonian can then be integrated and linked withthe digitized type specimen collections at other majorworld herbaria such as the New York Botanical Gardenthe Missouri Botanical Garden and the HarvardUniversity Herbaria However we soon realized that inmany cases the type specimens are not the best represen-tatives of variation in a species so we have broadenedour use of non-type collections as well in developing theimage library Second we are developing plant recogni-tion algorithms for comparing and ranking visual simi-larity in the recorded images of plant species Theserecognition algorithms are central to searching the imageportion of the digital collection of plant species and willbe joined with conventional search strategies in the tex-tual portion of the digital collection Third we are devel-oping a set of prototype devices with mobile user inter-faces to be tested and used in the field In this paper wewill describe our progress towards building a digital col-lection of the Smithsonianrsquos type specimens developingrecognition algorithms that can match an image of a leafto the species of plant from which it comes and design-ing user interfaces for electronic field guides

To start we are developing prototype electronic fieldguides for the flora of Plummers Island a small well-

studied island in the Potomac River Our prototype sys-tems contain multiple images for each of about 130species of plants on the island and should soon grow tocover all 200+ species currently recorded (Shetler amp alin press) Images of full specimens are available as wellas images of isolated leaves of each species Zoomableuser interfaces allow a user to browse these imageszooming in on ones of interest Visual recognition algo-rithms assist a botanist in locating the specimens that aremost relevant to identify the species of a plant Our sys-tem currently runs on small hand-held computers andTablet PCs We will describe the components of theseprototypes and also explain some of the future chal-lenges we anticipate if we are to provide botanists in thefield with access to all the resources that are now cur-rently available in the worldrsquos museums and herbaria

TYPE SPECIMEN DIGITAL COLLEC-TION

The first challenge in producing our electronic fieldguides is to create a digital collection covering all of theSmithsonianrsquos 85000 vascular plant type specimens Foreach type specimen the database should eventuallyinclude systematically acquired high-resolution digitalimages of the specimen textual descriptions links todecision trees images of live plants and 3D models

So far we have taken high resolution photographs ofapproximately 78000 of the Smithsonianrsquos type speci-mens as illustrated in Fig 1 At the beginning of ourproject specimens were photographed using a Phase OneLightPhase digital camera back on a Hasselblad 502 withan 80 mm lens This created an 18 MB color image with2000times3000 pixels More recently we have used a PhaseOne H20 camera back producing a 3600times5000 pixelimage The image is carried to the processing computerby IEEE 1394FireWire and is available to be processedinitially in the proprietary Capture One software thataccompanies the H20 The filename for each high-reso-lution TIF image is the unique value of the bar code thatis attached to the specimen The semi-processed imagesare accumulated in job batches in the software When thejob is done the images are finally batch processed inAdobe Photoshop and from these derivatives can begenerated for efficient access and transmission Low res-olution images of the type specimens are currently avail-able on the Smithsonianrsquos Botany web site at httpravenelsiedubotanytypes Full resolution images maybe obtained through ftp by contacting the CollectionsManager of the United States National Herbarium

In addition to images of type specimens it is ex-tremely useful to obtain images of isolated leaves of eachspecies of plant Images of isolated leaves are used in our


599

prototype systems in two ways First of all it is mucheasier to develop algorithms that judge the similarity ofplants represented by isolated leaves than it is to dealwith full specimens that may contain multiple overlap-ping leaves as well as branches While the appearance ofbranches or the arrangement of leaves on a branch mayprovide useful information about the identity of a plantthis information is difficult to extract automatically Theproblem of automatically segmenting an image of acomplex plant specimen to determine the shape of indi-vidual leaves and their arrangement is difficult and stillunsolved We are currently addressing these problemsbut for our prototype field guides we have simply beencollecting and photographing isolated leaves from allspecies of plants present on Plummers Island Secondlynot only are isolated leaves useful for automatic recogni-tion but they also make useful thumbnailsmdashsmallimages that a user can quickly scan when attempting tofind relevant information We will discuss this further ina following section

As our project progresses we plan to link other fea-tures to this core database such as other sample speci-mens (not types) of which there are 47 million in theSmithsonian collection live plant photographs and com-puter graphics models such as image-based or geometric

descriptions We also aim to enable new kinds of datacollection in which botanists in the field can record largenumbers of digital photos of plants along with informa-tion (accurate to the centimeter) about the location atwhich each photo and sample were taken

The full collection of 47 million plant specimensthat the Smithsonian manages provides a vast represen-tation of ecological diversity including as yet unidenti-fied species some of which may be endangered orextinct Furthermore it includes information on variationin structure within a species that can be key for visualidentification or computer-assisted recognition A poten-tial future application of our recognition algorithms is forautomatic classification of these specimens linking themto the corresponding type specimens and identifyingcandidates that are hard to classify requiring furtherattention from botanists In this way the visual and tex-tual search algorithms discussed in the next section couldbe used to help automatically discover new speciesamong the specimens preserved in the Smithsonianmdashsomething that would be extremely laborious withoutcomputer assistance

The most direct approach to acquisition and repre-sentation is to use images of the collected specimensHowever it will also be of interest to explore how


600

Fig 1 On the left our set-up at the Smithsonian for digitally photographing type specimens On the right an exampleimage of a type specimen

advances in computer graphics and modeling could buildricher representations for retrieval and display that cap-ture information about the 3-D structure of specimens Afirst step towards this might concentrate on creatingimage-based representations of the specimens buildingon previous work in computer graphics on image-basedrendering methods (Chen amp Williams 1993 McMillanamp Bishop 1995 Gortler amp al 1996 Levoy amp Hanrahan1996 Wood amp al 2000) The challenge for thisapproach will be to use a small number of images tokeep the task manageable while capturing a full modelof the specimen making it possible to create syntheticimages from any viewpoint One advantage is that spec-imens such as leaves are relatively flat and thereforehave a relatively simple geometry One could thereforestart with leaf specimens combining view-dependenttexture-mapping approaches (Debevec amp al 1996) withrecent work by us and others on estimating materialproperties from sparse sets of images by inverse tech-niques (Sato amp al 1997 Yu amp Malik 1998 Yu amp al1999 Boivin amp Gagalowicz 2001 Ramamoorthi ampHanrahan 2001)

Finally it would be very exciting if one could meas-ure geometric and photometric properties to create com-plete computer graphics models Geometric 3D informa-tion will allow for new recognition and matching algo-rithms that better compensate for variations in viewpointas well as provide more information for visual inspec-tion For example making a positive classification maydepend on the precise relief structure of the leaf or itsveins Also rotating the database models to a new view-point or manipulating the illumination to bring out fea-tures of the structure could provide new tools for classi-fication Furthermore this problem is of interest not justto botanists but also to those seeking to create realisticcomputer graphics images of outdoor scenes Whilerecent work in computer graphics (Prusinkiewicz amp al1994 Deussen amp al 1998) has produced stunning verycomplex outdoor images the focus has not been onbotanical accuracy

To achieve this one can make use of recent advancesin shape acquisition technology such as using laser rangescanning and volumetric reconstruction techniques(Curless amp Levoy 1996) to obtain geometric modelsOne can attempt to acquire accurate reflectance informa-tion or material properties for leavesmdashsomething thathas not been done previously To make this possible wehope to build on and extend recent advances in measure-ment technology such as image-based reflectance meas-urement techniques (Lu amp al 1998 Marschner amp al2000) Even accurate models of a few leaves acquired inthe lab might be useful for future approaches to recogni-tion that use photometric information Reflectance mod-els may also be of interest to botanists certainly research

on geometric and reflectance measurements would be ofimmense importance in computer graphics for synthesiz-ing realistic images of botanically accurate plants

VISUAL MATCHING OF ISOLATEDLEAVES

Our project has already produced a large collectionof digital images and we expect that in general thenumber of images and models of plants will grow to hugeproportions This represents a tremendous potential re-source It also represents a tremendous challenge be-cause we must find ways to allow a botanist in the fieldto navigate this data effectively finding images of speci-mens that are similar to and relevant to the identificationof a new specimen Current technology only allows forfully automatic judgments of specimen similarity that aresomewhat noisy so we are building systems in whichautomatic visual recognition algorithms interact withbotanists taking advantage of the strengths of machinesystems and human expertise

Many thorny problems of recognition such as seg-mentation and filtering our dataset to a reasonable sizecan be solved for us by user input We can rely on abotanist to take images of isolated leaves for matchingand to provide information such as the location and fam-ily of a specimen that can significantly reduce the num-ber of species that might match it With that startingpoint our system supplements text provided by the userby determining the visual similarity between a new sam-ple and known specimens Our problem is simplified bythe fact that we do not need to produce the right answerjust a number of useful suggestions

PRIOR WORK ON VISUAL SIMILAR-ITY OF ORGANISMS

There has been a large volume of work in the fieldsof computer vision and pattern recognition on judgingthe visual similarity of biological organisms Probablythe most studied instance of this problem is in the auto-matic recognition of individuals from images of theirfaces (see Zhao amp al 2003 for a recent review) Earlyapproaches to face recognition used representationsbased on face-specific features such as the eyes noseand mouth (Kanade 1973) While intuitive these ap-proaches had limited success for two reasons First thesefeatures are difficult to reliably compute automaticallySecond they lead to sparse representations that ignoreimportant but more subtle information Current ap-proaches to face recognition have achieved significantsuccess using much more general representations These


601

are informed by the general character of the object recog-nition problem but are not based on specific propertiesof the human face (eg Turk amp Pentland 1991 Lades ampal 1993) Some successful methods do make use of pri-or knowledge about the 3D shape of faces (eg Blanz ampVetter 2003) However successful approaches do not ne-cessarily rely on representations of faces that might seemto correspond to human descriptions of faces (eg a bignose or widely spaced eyes) Such systems achieve thebest current performance allowing effective retrieval offaces from databases of over a thousand images provid-ed that these are taken under controlled conditions (Phil-lips amp al 2002) This encourages us to explore the useof general purpose shape matching algorithms in plantidentification before attempting to compute descriptionsof plants that correspond to more traditional keys

There has also been work on automatic identificationof individuals within a (nonhuman) species For exam-ple biologists may track the movements of individual a-nimals to gather information about population sizes de-mographic rates habitat utilization and movement ratesand distances This is done using visual recognition incases in which mark-recapture methods are not practicalbecause of the stress they place on animals for exampleor the small size of some animals (eg salamanders)Some researchers have exploited distinctive visual traitsrecorded from direct observations and photographs suchas scars on cetaceans (Hammond 1990) or more generalshape characteristics of marine invertebrates (Hillman2003) pelage patterns in large cats (Kelley 2001) andmarkings on salamanders (Ravela amp Gamble 2004)

A number of studies have also used intensity patternsto identifying the species of animals and of some plantssuch as pollen or phytoplankton Gaston amp OrsquoNeill(2004) review a variety of these methods They primari-ly apply fairly general pattern recognition techniques tothe intensity patterns found on these organisms

There have also been several works that focusspecifically on identifying plants using leaves and thesehave generally concentrated on matching the shape ofleaves For example Abbasi amp al (1997) and Mokhtari-an amp Abbasi (2004) have worked on the problem of clas-sifying images of chrysanthemum leaves Their motiva-tion is to help automate the process of evaluating appli-cants to be registered as new chrysanthemum varietiesThe National Institute of Agricultural Botany in Britainhas registered 3000 varieties of chrysanthemums andmust test 300 applicants per year for distinctness Theycompare silhouettes of leaves using features based on thecurvature of the silhouette extracted at multiple scalesalso using other features that capture global aspects ofshape They experiment using a dataset containing tenimages each of twelve different varieties of chrysanthe-mums They achieve correct classification of a leaf up to

85 of the time and the correct variety is among the topthree choices of their algorithm over 98 of the timeWhile the number of varieties in their dataset is rathersmall their problem is difficult because of the close sim-ilarity between different varieties of chrysanthemums

In other work Im amp al (1998) derive a descriptionof leaf shapes as a straight line approximation connectingcurvature extrema although they do not report recogni-tion results only showing a few sample comparisonsSaitoh amp Kaneko (2000) use both shape and color fea-tures in a system that recognizes wild flowers using aneural net which is a general approach developed forpattern recognition and machine learning In a datasetrepresenting sixteen species of wildflowers they correct-ly identify a species over 95 of the time although per-formance drops drastically when only the shape of leavesare used Wang amp al (2003) develop a novel shape de-scription the centroid-contour distance which they com-bine with more standard global descriptions of shapeThey use this for recognition on a dataset containing1400 leaves from 140 species of Chinese medicinalplants This approach correctly recognizes a new leaf on-ly about 30 of the time and a correct match is from thesame species as one of the top 50 leaves retrieved onlyabout 50 of the time However this dataset appears tobe challenging and their system does outperform meth-ods based on curvature scale-space (Abbasi amp al 1997)and another method based on Fourier descriptors It issomewhat difficult to evaluate the success of these previ-ous systems because they use diverse datasets that arenot publicly available However it appears that goodsuccess can be achieved with datasets that contain fewspecies of plants but that good performance is muchmore difficult when a large number of species are used

In addition to these methods Soumlderkvist (2001) pro-vides a publicly available dataset (httpwwwisyliusecvlProjectsSweleafsamplepagehtml) containing 75leaves each from 15 different species of plants alongwith initial recognition results using simple geometricfeatures in which a recognition rate of 82 is reported

MEASURING SHAPE SIMILARITYThe core requirement of providing automatic assis-

tance to species identification is a method for judging thesimilarity of two plant specimens While there are manyways one can compare the visual similarity of plant spec-imens we have begun by focusing on the problem ofdetermining the similarity of the shape of two leaves thathave been well segmented from their background It willbe interesting in the future to combine this with othervisual information (eg of fruit or flowers) and metada-ta (eg describing the position of a leaf) We will now


602

discuss a novel approach that performs this task wellWhen we discuss our prototype user interfaces we willexplain how we achieve segmentation in practice andhow we use this measure of leaf similarity to aid inspecies identification

Our approach is motivated by the fact that leavespossess complex shapes with many concavities Manyapproaches to visual recognition attempt to deal withcomplex shapes by dividing an object into parts (egSiddiqi amp al 1999 Sebastian amp al 2004) Howeverleaves often lack an obvious part structure that fully cap-tures their shape Therefore we have instead introduceda new shape representation that reflects the part structureand concavities of a shape without explicitly dividing ashape into parts

Our representation is based on the inner-distancebetween two points on the boundary of a shape This isdefined as the length of the shortest path that connectsthe two points and that lies entirely inside the leaf Figure2 illustrates the inner-distance by showing a pair ofpoints (p q) on the boundary of three idealized leaf sil-houettes The Euclidean distance between the two pointsin (b) is more similar to the Euclidean distance betweenthe pair of points in (c) than those in (a) Shape descrip-tions based on this ordinary notion of distance will treat(b) as more similar to (c) In contrast the inner-distancebetween the pair of points in (b) is more similar to thosein (a) than in (c) As a consequence the inner-distance-based shape descriptors can distinguish the three shapesmore effectively

Using the inner-distance we build a description ofthe relationship between a boundary point and a set ofother points sampled along the leaf boundary This con-sists of a histogram of the distances to all other boundarypoints as well as the initial direction of the shortest innerpath to each point This descriptor is closely related to

the shape context (Belongie amp al 2002) using the inner-distance instead of the Euclidean distance We thereforecall our method the Inner-Distance Shape Context(IDSC)

Once a descriptor is computed for each point on theboundary of the leaf we compare two leaves by findinga correspondence between points on the two leaves andmeasuring the overall similarity of all correspondingpoints Using a standard computer science algorithmcalled dynamic programming we can efficiently find thebest possible matching between points on the two leafboundaries that still preserves the order of points alongthe boundary More details of this algorithm can be foundin Ling amp Jacobs (2005)

We have tested this algorithm on a set of isolatedleaves used in one of our prototype systems This test setcontains images of 343 leaves from 93 different speciesof plants found on Plummers Island and is available athttpwwwcsumdedu~hblingResearchdataSI-93zip In the experiment 187 images are used as trainingexamples that the system uses as its prior informationabout the shape of that species The remaining 156 im-ages are used as test images For each test image wemeasure how often a leaf from the correct species isamong the most similar leaves in the training setInformation about the effectiveness of a retrieval mecha-nism can be summarized in the Receiver OperatingCharacteristics (ROC) curve (Fig 3) This measures howthe probability of retrieving the correct match increaseswith the number of candidate matches produced by thesystem The horizontal axis shows the number of speciesretrieved and the vertical axis shows the percentage oftimes that the correct species is among these

The figure compares our method to ordinary shapecontexts (SC) and to Fourier Descriptors computed usingthe discrete Fourier transform (DFT) FourierDescriptors are a standard approach to shape descriptionin which a contour is described as a complex functionand the low-frequency components of its Fourier trans-form are used to capture its shape (see eg Lestrel1997) The figure shows that our new method (IDSC)performs best although IDSC and SC perform similarlyonce twenty or more species of plants are retrievedThese results show that if we use IDSC to present a userwith several potential matching species to a new leafspecimen the correct species will be among these choic-es well over 95 of the time Note that our results areeither much more accurate than or use a much largerdataset than the prior work that we have describedHowever it is difficult to compare results obtained ondifferent datasets since they can vary widely in difficul-ty depending on the similarity of plants used We havenot yet been able to compare IDSC to all prior methodsfor species identification on a common dataset However


603

Fig 2 This figure shows three compound leaves where(a) and (b) are from the same species which is differentfrom that of (c) The lengths of the dotted line segmentsdenote the Euclidean distances between two points pand q The solid lines show the shortest paths between pand q The inner-distances are defined as the length ofthese shortest paths The Euclidean distance between pand q in (b) is more similar to that in (c) than in (a) Incontrast the inner-distance between p and q in (b) ismore similar to that in (a) than in (c)

Ling amp Jacobs (2005) compare IDSC to ten other shapecomparison methods on a variety of commonly used testsets of non-leaf shapes IDSC performs better than thecompeting methods in every case

In addition to these experiments we have also testedour algorithm on Soumlderkvistrsquos dataset The top speciesmatched to a leaf by our algorithm is correct 94 of thetime compared to Soumlderkvistrsquos reported results of 82Our own implementations of Fourier Descriptors and amethod using Shape Contexts achieve 90 and 88recognition performance respectively

While these experimental results are encouragingthey should be viewed with some caution First of all inour current test set all leaves from a single species werecollected at the same time from the same plantTherefore data does not reflect the possible variationsthat can occur as leaves mature or between differentplants growing in different regions We are in the processof collecting additional specimens now which will helpus to test our algorithm in the face of some of this varia-tion Second for many environments it will be necessaryto perform retrieval when many more species of plantsare possible candidates for matching We anticipate thatin many cases it will be necessary to supplement shapematching with information about the intensity patternswithin leaves especially of the venation While impor-tant these features are more difficult to extract automat-ically and will pose a significant challenge in futurework

We also plan in future work to evaluate our algo-rithm for use in matching an isolated leaf to a full plantspecimen This would be helpful since it may not be fea-sible for us to collect new specimens for every species ofplant in the type specimen collection of the United StatesNational Herbarium One possible solution is to extractisolated leaves from the type specimens and use thesefor matching This might be done manually with a gooddeal of work and the aid of some automatic tools We arealso exploring methods for automatically segmentingleaves in images of type specimens although this is avery challenging problem due to the extreme clutter inmany of these images

PROTOTYPE ELECTRONIC FIELDGUIDES

We have integrated our approach to visual similaritywith existing tools for image browsing into several pro-totype Electronic Field Guides (EFGs) Our EFGs cur-rently contain information about 130 species of plantsfound on Plummers Island

While we envision that a field guide will ultimatelyreside on devices no larger than current PDAs we want-

ed to avoid the short-term constraints of implementingfor existing PDAs with their low-resolution displayssmall memories underpowered CPUs and limited oper-ating systems For this reason our initial handheld pro-totype was implemented on a slightly larger but far morepowerful ultraportable hand-held computer sometimesreferred to as a ldquohandtoprdquo Our entire initial system runson a Sony VAIO U750 running Windows XPProfessional This device measures 657times425times103weighs 12 lbs and includes a 11 GHz Mobile PentiumM processor 512 MB Ram a 5 800times600 resolutioncolor sunlight-readable display a touch-sensitive displayoverlay and 20 GB hard drive

Our initial prototype was based on PhotoMesa(Bederson 2001) which is a zoomable user interface Itshows the thumbnails of images arranged in 2D array Asa user mouses on an image the image is shown at asomewhat larger size The user can zoom inout using themouse (leftright) clicks If the user wants to see moreimages of a particular species she can double click withthe left mouse button Thus she can control the informa-tion she wants to see at any instant These mouse-basedcontrols make browsing and navigation easier

We added some new features to the browsing capa-bilities offered by PhotoMesa

Similarity based clustering and display ofdata mdash Species are clustered based on the similaritybetween images of isolated leaves from these species Todo this we use a variation on the k-means clustering algo-rithm (see eg Forsyth amp Ponce 2003) in which indi-vidual leaves are used as cluster centers The images aredisplayed so that similar images are displayed as groupsThis should help the user to better isolate the group towhich the query image belongs Once the correct group


604

Fig 3 ROC curves describing the performance of ournew method (IDSC) in contrast with two other methods(SC DFT = discrete Fourier transform)

is determined she can zoom into that group to get moreinformation

Visual search mdash The user can provide a queryimage which is matched with the images in the databaseusing the IDSC method introduced above In a completesystem a user will be able to photograph a leaf collectedin the field and use this as input to the system In this ini-tial prototype images we have taken of isolated leavesare provided as input Example leaves from the 20 bestmatching species are then displayed to the user

Text search mdash The user can see the images for aparticular genus or species by typing the names Partialnames are accepted

The prototype database consists of large specimensincluding Type Specimens and isolated leaf images (Fig5) At present the database has over 1500 isolated leavesfrom 130 species and approximately 400 larger speci-men images for 70 species

The shape matching algorithm requires the contourof leaves as input Since leaves are photographed on a

plain background extracting their contour reliably is nottoo difficult although this process is simplified if care istaken to avoid shadows in the image We obtain the con-tour also by using k-means clustering to divide the imageinto two regions based on color We use the central andthe border portions of the image as initial estimates of theleaf and background colors We then use some simplemorphological operations to fill in small holes (see egHaralick amp Shapiro 1992) The input and output of thisstep are shown in Fig 6

We have performed user studies to evaluate theeffectiveness of the different features of this prototypesystem using 21 volunteers from the Department ofBotany at the Smithsonian Institution In these studiesusers were presented with a query image showing an iso-lated leaf from Plummers Island and asked to identifythe species of the leaf using three versions of our systemIn some queries the users were given the basic featuresof PhotoMesa described above with leaves placed ran-domly A second version placed leaves using clusteringA third version displayed the results of visual search tothe user

A complete description of our results is presented byAgarwal (2005) Here we briefly summarize theseresults With random placement users identified 71 ofthe query leaves correctly in an average time of 30 sec-onds With clustering users found the correct species84 of the time in an average time of 29 seconds Withvisual search users were correct 85 of the time with anaverage time of 22 seconds The superior performance ofsubjects using visual search was statistically significantboth in terms of time and accuracy Subjects also per-formed better with clustering than with random place-ment but this difference was not statistically significantIn addition users reported higher satisfaction with thesystem using visual search next highest with clusteringand the least satisfaction with random placement Allthese differences in satisfaction were statistically signifi-cant

Our results suggest that visual search can produce asignificantly better retrieval system and that clusteringmay also produce a system that is easier to use comparedto one in which images are placed randomly Howevervisual search will require substantially more overheadthan the other methods since a user must photograph aleaf in order to use visual search to help identify itsspecies The other features of the EFG can be used with-out photographic input We believe that these results arepromising but that to measure the true impact of thesemethods it is essential to experiment with larger datasetsand with live field tests in which a botanist uses the sys-tem to identify plants in the field A difference of a fewseconds to identify a plant is probably not important inpractice but the differences we observe for this relative-


605

Fig 4 Screenshots from our initial EFG prototype withrandom placement (top) and placement based on cluste-ring (bottom)

ly small dataset may be magnified in systems containinginformation about hundreds of species of plants We planto expand our experiments in these directions

More recently we have developed a second proto-type based on a Tablet PC tablet-based system whichincorporates lessons learned from our user studies andinitial prototype to provide a user interface that can beeffectively deployed in the field (Fig 7) The systemhardware incorporates an IBM Thinkpad X41 Tablet PC(shown in the figure but recently replaced with a MotionComputing LE 1600 Tablet PC with daylight-readabledisplay) a Delorme Earthmate Bluetooth and a NikonCoolPix P1 camera that transmits images wirelessly tothe tablet PC The user interface comprises five screensaccessible through a tabbed interface much like a tabbedset of file folders browse current sample search resultshistory and help

The browse tab represents the entire visual leaf data-base in a zoomable user interface developed usingPiccolo (Bederson amp al 2004) The species are present-ed as an ordered collection based on the clustering algo-rithm described earlier Each leaf node in the visual data-base contains individual leaf images reference text anddigital images of specimen vouchers for that species Thecollection can be browsed by looking at sections of thecollection or at individual leaves

The next tab displays the current sample leaf Takinga picture of a leaf with the digital camera or selecting anearlier photo from the history list (described later) placesthe image of the leaf in this tab Each picture is immedi-

ately placed in a database along with relevant contextu-al information such as time collector and GPS coordi-nates The black and white outline of the segmentedimage which is used to identify the leaf is displayednext to the original image as feedback to the botanistregarding the quality of the photo If the silhouette looksbroken or incomplete a replacement photo can be takenA button at the bottom of the screen can be used to dis-play the search results

The third tab presents the results of a visual searchusing the current sample as shown in Fig 7 Leaves aredisplayed in an ordered collection with the best matchespresented at the top Each represented species can beinspected in the same way that leaves in the browser canbe inspected including all voucher images for a givenspecies If the botanist feels they have found a speciesmatch pressing a button with a finger or stylus associatesthat species with the current sample in the database

The fourth tab provides a visual history of all thesamples that have been collected using the system Thesamples in this leaf collection can be enlarged for inspec-tion and the search results from a given sample can beretrieved The last tab provides a simple help referencefor the system

Our current visual search algorithm returns anordered set of the top ten possible matches to a leaf Inthe future it will be interesting to explore how the usercan help narrow the identification further This couldinvolve a collaborative dialogue in which the systemmight suggest that the user provide additional input datasuch as one or more new images (perhaps taken from anexplicitly suggested vantage point determined from ananalysis of the originally submitted image) In additionit would be interesting to develop ways for users to spec-ify portions of a 2D image (and later of a 3D model) onwhich they would like the system to focus its attentionand for the system to specify regions on which the usershould concentrate Here we will rely on our work onautomating the design and unambiguous layout of textu-al and graphical annotations for 2D and 3D imagery


606

Fig 5 A sample photograph of an isolated leaf

Fig 6 Isolated leaf before and after preprocessing

(Bell amp Feiner 2000 Bell amp al 2001) which we havealready used with both head-worn and hand-held dis-plays

WEARABLE COMPUTINGTo complement our hand-held prototypes we are

exploring the use of head-tracked see-through head-worn displays which free the userrsquos hands and provideinformation that is overlaid on and integrated with theuserrsquos view of the real world Here we are building onour earlier work on wearable backpack-based mobileaugmented reality systems (eg Feiner amp al 1997Houmlllerer amp al 1999b) One focus is on addressing theeffective display of potentially large amounts of infor-mation in geospatial context including spatializedrecords of field observations during the current and pre-vious visits to a site and 3D models Another focus isfacilitating the search and comparison task by presentingresults alongside the specimen

Our earlier backpack systems have used relativelylarge and bulky displays (eg a Sony LDI-D100B stereocolor display with the size and appearance of a pair of skigoggles and a MicroVision Nomad retinal scanning dis-play) In contrast we are currently experimenting with aKonica Minolta prototype ldquoforgettablerdquo display (Kasai ampal 2000) which is built into a pair of conventional eye-glasses a commercially available MicroOptical ldquoclip-onrdquo display that attaches to conventional eyeglasses aLiteye LE-500 monocular see-through display and sev-eral other commercial and prototype head-worn displaysIn addition to the touch-sensitive display overlays thatare part of our hand-held prototypes we are also inter-ested in the use of speech input and output to exploit thehands-free potential of head-worn displays and smallermanual interaction devices that do not require the userrsquos

visual attention (Blaskoacute amp Feiner 2005)We have been developing two mobile augmented

reality user interfaces that use head-tracked head-worndisplays (White amp al 2006) One user interface providesa camera-tracked tangible augmented reality in which theresults of the visual search are displayed next to the phys-ical leaf The results consist of a set of virtual vouchersthat can be physically manipulated for inspection andcomparison In the second user interface the results of avisual search query are displayed in a row in front of theuser floating in space and centered in the userrsquos field ofview These results can be manipulated for inspectionand comparison by using head-movement alone trackedby an inertial sensor leaving the hands free for othertasks

To support geographic position tracking in our hand-held and head-worn prototypes we have used theDeLorme Earthmate a small WAAS-capable GPSreceiver WAAS (Wide Area Augmentation System) is aFederal Aviation Administration and Department ofTransportation initiative that generates and broadcastsGPS differential correction signals through a set of geo-stationary satellites These error corrections make possi-ble positional accuracy of less than three meters in NorthAmerica While regular GPS would be more than ade-quate to create the collection notes for an individualspecimen and assist in narrowing the search to winnowout species that are not known to live in the userrsquos loca-tion we are excited about the possibility of using theincreased accuracy of WAAS-capable GPS to more accu-rately localize individual living specimens relative toeach other

Future prototypes of our electronic field guide couldtake advantage of even higher accuracy real-time-kine-matic GPS (RTK GPS) to support centimeter-level local-ization as used in our current outdoor backpack-drivenmobile augmented reality systems Coupled with theinertial head-orientation tracker that we already exploitin our augmented reality systems this would allow us tooverlay on the userrsquos view of the environment relevantcurrent and historical data collected at the site with eachitem positioned relative to where it was collected build-ing on our earlier work on annotating outdoor sites withhead-tracked overlaid graphics and sound (Houmlllerer ampal 1999a Guumlven amp Feiner 2003 Benko amp al 2004)Further down the road we will be experimenting with anArc Second Constellation 3DI wide-area outdoor 6DOFtracker (+- 004 accuracy) to track a camera and displayover a significant area This could make it possible toassemble sets of images for use in creating 3D models inthe field of both individual plants and larger portions ofa local ecosystem

In our current prototypes all information is residenton the disk of a mobile computer This may not be possi-


607

Fig 7 Search results shown by our second prototyperunning on a Tablet PC

ble for future prototypes that make use of more data ordata that might not be readily downloaded These proto-types and their user interfaces should be built so that thedatabases can reside entirely on the remote server or bepartially cached and processed on the mobile system Wealso note that in field trials even low-bandwidth world-wide-web connectivity may not be available In this casea local laptop with greater storage and computationcapabilities than the hand-held or wearable computercould be used to host the repository (or the necessarysubset) and run the recognition algorithms communicat-ing with the hand-held or wearable computer through alocal wireless network

CONCLUSIONSNew technology is making it possible to create vast

digital collections of botanical specimens This offersimportant potential advantages to botanists includinggreater accessibility and the potential to use automaticstrategies to quickly guide users to the most relevantinformation We have sketched an admittedly ambitiousplan for building and using these collections and havedescribed our initial results

The work that we have accomplished thus far helpsunderscore three key challenges The first challenge is tobuild digital documents that capture information avail-able in libraries herbaria and museums This includesthe well-understood task of building large collections ofhigh resolution images and the still open problems ofhow best to capture visual information about 3D speci-mens The second challenge is to build effective retrievalmethods based on visual and textual similarity Whilethis is a very difficult research problem we haveimproved on existing methods for visual search and pro-vided evidence that these methods may provide usefulsupport to botanists We have done this by building pro-totype systems that combine visual similarity measureswith image browsing capabilities This provides aretrieval system that doesnrsquot solve the problem of speciesidentification but rather aims to help a botanist to solvethis problem more efficiently While our work to date hasfocused largely on leaf shape we intend to consider othermeasures of visual similarity such as leaf venation tex-ture and even multispectral color Furthermore we havejust begun to investigate how visual search can be com-bined with textual search with an aim toward leveragingthe existing taxonomic literature Finally in addition tothe image retrieval methods present in our prototype sys-tems we have discussed some of the ways that futurework in wearable computing may allow us to build sys-tems that make it possible for a botanist to use and col-lect even more information in the field

ACKNOWLEDGEMENTSThis work was funded in part by National Science Foundation

Grant IIS-03-25867 An Electronic Field Guide Plant Explorationand Discovery in the 21st Century and a gift from MicrosoftResearch

LITERATURE CITEDAbbasi S Mokhtarian F amp Kittler J 1997 Reliable clas-

sification of chrysanthemum leaves through curvaturescale space Pp 284ndash295 in Romeny B M Florack LKoenderink J J amp Viergever M A (eds) Proceedingsof the First international Conference on Scale-Space the-ory in Computer Vision Springer-Verlag London

Agarwal G 2005 Presenting Visual Information to the UserCombining Computer Vision and Interface Design MScsThesis Univ Maryland Maryland

Bederson B 2001 PhotoMesa A zoomable image browserusing quantum treemaps and bubblemaps Pp 71ndash80 inUIST 2001 Proceedings of the 14th Annual ACMSymposium on User Interface Software and TechnologyNew York ACM Press New York

Bederson B Grosjean J amp Meyer J 2004 Toolkit designfor interactive structured graphics IEEE Trans SoftwareEngineering 30 535ndash546

Belongie S Malik J amp Puzicha J 2002 Shape matchingand object recognition using shape context IEEE TransPAMI 24 509ndash522

Bell B amp Feiner S 2000 Dynamic space management foruser interfaces Pp 239ndash248 in Proceedings of the ACMSymposium on User Interface Software and Technology(CHI Letters vol 2 no 2) San Diego ACM Press NewYork

Bell B Feiner S amp Houmlllerer T 2001 View management forvirtual and augmented reality Pp 101ndash110 inProceedings of the ACM Symposium on User InterfaceSoftware and Technology (CHI Letters vol 3 no 3)ACM Press New York

Benko H Ishak E amp Feiner S 2004 Collaborative mixedreality visualization of an archaeological excavation Pp132ndash140 in (eds) Proceedings of IEEE and ACMInternational Symposium on Mixed and AugmentedReality Arlington IEEE Computer Society Press LosAlamitos

Bisby F A 2000 The quiet revolution biodiversity informat-ics and the Internet Science 289 2309ndash2312

Blanz V amp Vetter T 2003 Face recognition based on fittinga 3D morphable model IEEE Trans PAMI 25 1063ndash1074

Blaskoacute G amp Feiner S 2005 Input devices and interactiontechniques to minimize visual feedback requirements inaugmented and virtual reality In Proceedings of HCIInternational Las Vegas NV July 22ndash27 LawrenceErlbaum Associates Mahwah [published on CDROMonly]

Boivin S amp Gagalowicz A 2001 Image-based rendering ofdiffuse specular and glossy surfaces from a single imageSIGGRAPH 01 107ndash116

Chen S amp Williams L 1993 View interpolation for image


608

synthesis SIGGRAPH 93 279ndash288Curless B amp Levoy M 1996 A volumetric method for build-

ing complex models from range images SIGGRAPH 96303ndash312

CyberTracker Conservation 2005 Website for CyberTrackerSoftware and CyberTracker Conservation httpwwwcybertrackerorg

Debevec P Taylor C amp Malik J 1996 Modeling and ren-dering architecture from photographs A hybrid geometryand image-based approach SIGGRAPH 96 11ndash20

Deussen O Hanrahan P Lintermann B Mech RPharr M amp Prusinkiewicz P 1998 Realistic modelingand rendering of plant ecosystems SIGGRAPH 98275ndash286

Edwards J L Lane M A amp Nielsen E S 2000 Intero-perability of biodiversity databases biodiversity informa-tion on every desktop Science 289 2312ndash2314

Edwards M amp Morse D R 1995 The potential for com-puter-aided identification in biodiversity research TrendsEcol Evol 10 153ndash158

Feiner S MacIntyre B Houmlllerer T amp Webster A 1997 Atouring machine Prototyping 3D mobile augmented reali-ty systems for exploring the urban environment PersTechnol 1 208ndash217

Forsyth D amp Ponce J 2003 Computer vision a modernapproach Prentice Hall Upper Saddle River

Gaston K amp OrsquoNeill M 2004 Automated species identifi-cation why not Phil Trans Royal Soc B Biol Sci 359655ndash667

Gortler S Grzeszczuk R Szeliski R amp Cohen M 1996The lumigraph SIGGRAPH 96 43ndash54

Govaerts R 2001 How many species of seed plants arethere Taxon 50 1085ndash1090

Guumlven S amp Feiner S 2003 A hypermedia authoring tool foraugmented and virtual reality New Rev HypermedMultimed 9 89ndash116

Hammond P S Mizroch S A amp Donavan G P 1990Individual Recognition of Cetaceans Use of Photo-Identi-fication and Other Techniques to Estimation PopulationParameters International Whaling Commission Cam-bridge Massachusetts [Special issue 12]

Haralick R M amp Shapiro L G 1992 Computer and RobotVision Addison-Wesley Reading

Haritaoglu I 2001 InfoScope Link from real world to digi-tal information space Pp 247ndash255 in Abowd G DBrumitt B amp Shafer S (eds) Proceedings of Ubicomp2001 (Third International Conference on UbiquitousComputing) Atlanta GA September 30ndashOctober 2Springer Berlin

Heidorn P B 2001 A tool for multipurpose use of onlineflora and fauna The Biological Information Browsing En-vironment (BIBE) First Monday 6 2 httpfirstmondayorgissuesissue6_2heidornindexhtml

Hillman G R Wursig B Gailey G A Kehtarnavaz NDorbyshevsky A Araabi B N Tagare H D ampWeller D W 2003 Computer-assisted photo-identifica-tion of individual marine vertebrates a multi-species sys-tem Aquatic Animals 29 117ndash123

Houmlllerer T Feiner S amp Pavlik J 1999a Situated docu-mentaries embedding multimedia presentations in the realworld Pp 79ndash86 in Proceedings of IEEE InternationalSymposium on Wearable Computers San Francisco

California October 18ndash19 IEEE Computer Society PressLos Alamitos

Houmlllerer T Feiner S Terauchi T Rashid G amp HallawayD 1999b Exploring MARS developing indoor and out-door user interfaces to a mobile augmented realty systemComputers Graphics 23 779ndash785

Im C Nishida H amp Kunii T L 1998 Recognizing plantspecies by leaf shapesmdasha case study of the Acer familyInt Conf Pattern Recognition 2 1171ndash1173

Kanade T 1973 Picture Processing by Computer Complexand Recognition of Human Faces Department ofInformation Science Kyoto University Kyoto

Kasai I Tanijiri Y Endo T amp Ueda H 2000 A forget-table near eye display Pp 115ndash118 in Proceedings ofIEEE International Symposium on Wearable ComputersOctober 16-17 Atlanta IEEE Computer Society PressLos Alamitos

Kelly M J 2001 Computer-aided photograph matching instudies using individual identification an example fromSerengeti cheetahs J Mammol 82 440ndash449

Kress W J 2002 Globalization and botanical inventory Pp6ndash8 in Watanabe T Takano A Bista M S amp Saiju HK (eds) Proceedings of the Nepal-Japan JointSymposium on ldquoThe Himalayan plants Can they saveusrdquo Kathmandu Nepal Society for the Conservation andDevelopment of Himalayan Medicinal Resources Tokyo

Kress W J 2004 Paper floras how long will they last Areview of Flowering Plants of the Neotropics Amer JBot 91 2124ndash2127

Kress W J amp Krupnick G A 2005 Documenting and con-serving plant diversity the future Pp 313ndash318 inKrupnick G A amp Kress W J (eds) Plant Conservationa Natural History Approach Univ Chicago PressChicago

Lades M Vortbruumlggen J C Buhmann J Lange J vonder Malsburg C Wuumlrtz R P amp Konen W 1993Distortion invariant object recognition IEEE TransComputers 42 300ndash311

Lestrel P 1997 Fourier Descriptors and their Applications inBiology Cambridge Univ Press Cambridge

Levoy M amp Hanrahan P 1996 Light field rendering SIG-GRAPH 96 31ndash42

Ling H amp Jacobs D 2005 Using the inner-distance for clas-sification of articulated shapes Pp 719ndash726 in IEEEConf on Comp Vis and Pattern Recognition vol 2 IEEEComputer Society Press Los Alamitos

Lu R Koenderink J amp Kappers A 1998 Optical proper-ties (bidirectional reflection distribution functions) of vel-vet Appl Opt 37 5974ndash5984

Marschner S Westin S Lafortune E amp Torrance K2000 Image-based BRDF measurement Appl Opt 392592ndash2600

McMillan L amp Bishop G 1995 Plenoptic modeling animage-based rendering system SIGGRAPH 95 39ndash46

Mokhtarian F amp Abbasi S 2004 Matching shapes withself-intersections application to leaf classification IEEETransactions on Image Processing 13 653ndash661

Pankhurst R J 1991 Practical Taxonomic ComputingCambridge Univ Press Cambridge

Pascoe J 1998 Adding generic contextual capabilities towearable computers Pp 92ndash99 in Proceedings of IEEEInternational Symposium on Wearable Computers


609

Pittsburgh PA October 19ndash20 IEEE Computer SocietyPress Los Alamitos

Phillips P Grother P Michaels R Blackburn DTabassi E amp Bone M 2003 Face Recognition VendorTest 2002 NISTIR 6965 National Institute of StandardsTechnology Gaithersburg

Prusinkiewicz P James M amp Mech R 1994 Synthetictopiary SIGGRAPH 94 351ndash358

Ramamoorthi R amp Hanrahan P 2001 A signal-processingframework for inverse rendering SIGGRAPH 01117ndash128

Ravela S amp Gamble L 2004 On recognizing individualsalamanders Pp 742ndash747 in Proceedings of Asian Con-ference on Computer Vision Ki-Sang Hong and ZhengyouZhang Ed Jeju Korea Asian Federation of ComputerVision Societies Jeju

Saitoh T amp Kaneko T 2000 Automatic recognition of wildflowers Int Conf Pattern Recognition 2 2507ndash2510

Sato Y Wheeler M D amp Ikeuchi K 1997 Object shapeand reflectance modeling from observation SIGGRAPH97 379ndash388

Scotland R W amp Wortley A H 2003 How many species ofseed plants are there Taxon 52101ndash104

Sebastian T Klein P amp Kimia B 2004 Recognition ofshapes by editing their shock graphs IEEE Trans PAMI26 550ndash571

Shetler S G Orli S S Beyersdorfer M amp Wells E F Inpress Checklist of the vascular plants of Plummers IslandMaryland Bull Biol Soc Wash

Siddiqi K Shokoufandeh A Dickinson S amp Zucker S1999 Shock graphs and shape matching Int J ComputerVision 3513ndash32

Soumlderkvist O 2001 Computer Vision Classification of Leavesfrom Swedish Trees MSc Thesis Linkoumlping UnivLinkoumlping

Stevenson R D Haber W A amp Morriss R A 2003 Elec-tronic field guides and user communities in the eco-infor-matics revolution Cons Ecol 7 3 httpwwwconsecolorgvol7iss1art3

Turk M amp Pentland A 1991 Eigen faces for recognition JCognitive Neurosci 3 71ndash86

Wang Z Chi W amp Feng D 2003 Shape based leaf imageretrieval IEE proc Vision Image and Signal Processing150 34ndash43

White S Feiner S amp Kopylec J 2006 Virtual vouchersPrototyping a mobile augmented reality user interface forbotanical species identification Pp 119ndash126 in Proc3DUI (IEEE Symp on 3D User Interfaces) Alexandria25ndash26 Mar 2006 IEEE Computer Society Press LosAlamitos

Wilson E O 2000 A global biodiversity map Science 2892279

Wilson E O 2003 The encyclopedia of life Trends EcolEvol 18 77ndash80

Wood D Azuma D Aldinger K Curless B DuchampT Salesin D amp Stuetzle W 2000 Surface light fieldsfor 3D photography SIGGRAPH 00 287ndash296

Yu Y Debevec P Malik J amp Hawkins T 1999 Inverseglobal illumination Recovering reflectance models of realscenes from photographs SIGGRAPH 99 215ndash224

Yu Y amp Malik J 1998 Recovering photometric properties ofarchitectural scenes from photographs SIGGRAPH 98

207ndash218Zhang J Chen X Yang J amp Waibel A 2002 A PDA-

based sign translator Pp 217ndash222 in Proceedings of In-ternational Conference on Multimodal Interfaces Pitts-burgh ACM Press New York

Zhao W Chellappa R Phillips P J amp Rosenfeld A2003 Face recognition a literature survey ACM Com-puting Surveys 35 399ndash458


610

through the partnership of natural history biologistscomputer scientists nanotechnologists and bioinfor-maticists (Wilson 2003 Kress 2004)

Accelerating the collection and cataloguing of newspecimens in the field must be a priority Augmentingthis task is critical for the future documentation of biodi-versity particularly as the race narrows between speciesdiscovery and species lost due to habitat destructionNew technologies are now being developed to greatlyfacilitate the coupling of field work in remote locationswith ready access to and utilization of data about plantsthat already exist as databases in biodiversity institutionsSpecifically a taxonomist who is on a field expeditionshould be able to easily access via wireless communica-tion through a laptop computer or high-end PDA criticalcomparative information on plant species that wouldallow himher to (1) quickly identify the plant in questionthrough electronic keys andor character recognition rou-tines (2) determine if the plant is new to science (3)ascertain what information if any currently exists aboutthis taxon (eg descriptions distributions photographsherbarium and spirit-fixed specimens living materialDNA tissue samples and sequences etc) (4) determinewhat additional data should be recorded (eg colors tex-tures measurements etc) and (5) instantaneously queryand provide information to international taxonomic spe-cialists about the plant Providing these data directly andeffectively to field taxonomists and collectors wouldgreatly accelerate the inventory of plants throughout theworld and greatly facilitate their protection and conser-vation (Kress 2004)

This technological vision for the future of taxonomyshould not trivialize the magnitude of the task before usIt is estimated that as many as 10 million species ofplants animals and microorganisms currently inhabit theplanet Earth but we have so far identified and describedless than 18 million of them (Wilson 2000) Scientistshave labored for nearly three centuries on systematicefforts to find identify name and make voucher speci-mens of new species This process of identification andnaming requires extremely specialized knowledge Atpresent over three billion preserved specimens arehoused in natural history museums herbaria and botan-ic gardens with a tremendous amount of informationcontained within each specimen Currently taxonomistson exploratory expeditions make new collections ofunknown plants and animals that they then bring back totheir home institutions to study and describe Usuallymany years of tedious work go by during which time sci-entists make comparisons to previous literature recordsand type specimens before the taxonomic status of eachnew taxon is determined

In the plant world estimates of the number of spe-cies currently present on Earth range from 220000 to o-

ver 420000 (Govaerts 2001 Scotland amp Wortley 2003)By extrapolation from what we have already describedand what we estimate to be present it is possible that atleast 10 percent of all vascular plants are still to be dis-covered and described (J Kress amp E Farr unpubl) Themajority of what is known about plant diversity existsprimarily as written descriptions that refer back to phys-ical specimens housed in herbaria Access to these spec-imens is often inefficient and requires that an inquiringscientist knows what heshe is looking for beforehandCurrently if a botanist in the field collects a specimenand wants to determine if it is a new species the botanistmust examine physical samples from the type specimencollections at a significant number of herbaria Althoughthis was the standard mode of operation for decadessuch arrangement is problematic for a number of reasonsFirst type specimens are scattered in hundreds of her-baria around the world Second the process of request-ing approving shipping receiving and returning typespecimens is extremely time-consuming both for theinquiring scientists and for the host herbarium It canoften take many months to complete a loan during whichtime other botanists will not have access to the loanedspecimens Third type specimens are often incompleteand even a thorough examination of the relevant typesmay not be sufficient to determine the status of a newspecimen gathered in the field New more efficientmethods for accessing these specimens and better sys-tems for plant identification are needed

Electronic keys and field guides two modern solu-tions for identifying plants have been available sincecomputers began processing data on morphological char-acteristics (Pankhurst 1991 Edwards amp Morse 1995Stevenson amp al 2003) In the simplest case of electron-ic field guides standard word-based field keys usingdescriptive couplets have been enhanced with colorimages and turned into electronic files to make identifi-cation of known taxa easier and faster More sophisticat-ed versions include electronic keys created through char-acter databases (eg Delta delta-intkeycom Lucidwwwlucidcentralorg) Some of these electronic fieldguides are now available on-line or for downloading ontoPDAs (eg Heidorn 2001 OpenKey httpwww3isrluiucedu~pheidorncgi-binschoolfieldguidecgi) whileothers are being developed as active websites that cancontinually be revised and updated (eg Flora of theHawaiian Islands httpravenelsiedubotanypacificislandbiodiversityhawaiianfloraindexhtm)

Somewhat further afield previous work has alsoaddressed the use of GPS-equipped PDAs to record in-field observations of animal tracks and behavior (egPascoe 1998 CyberTracker Conservation 2005) butwithout camera input or recognition (except insofar asthe skilled user is able to recognize on her own an animal


598











599







600












601











602










603














604












605











606













607























608








































609


























610











599







600












601











602










603














604












605











606













607























608








































609


























610







600












601











602










603














604












605











606













607























608








































609


























610











601











602










603














604












605











606













607























608








































609


























610











602










603














604












605











606













607























608








































609


























610










603














604












605











606













607























608








































609


























610













604












605











606













607























608








































609


























610











605











606













607























608








































609


























610










606













607























608








































609


























610











607























608








































609


























610






















608








































609


























610








































609


























610


























610

first steps toward an electronic field guide for plants

Documents