Marine Pollution Bulletin 62 (2011) 1377–1380

Contents lists available at ScienceDirect

Marine Pollution Bulletin

journal homepage: www.elsevier .com/ locate /marpolbul

Editorial

The problem with indices

We are writing this Editorial after listening to an index-ladengraduate student presentation at a scientific conference, in whichit was proposed that new indices be developed and compared toexisting indices. At the end we both stood up independently andsuggested that this was not a useful exercise. But new papers arestill being published proposing and using new indices. Clearly thisstudent is not the only one confused by what we would call badscientific practice. Credible scientists should not be developing orrelying on single number representations of complex data. Andthey should not be misleading non-scientists that this is appropri-ate or even useful.

Indices are appealing because they can be used to reduce com-plex data to single numbers, which seem easy to understand. Butthat is not biological or environmental reality, which is rarely 1-dimensional. At best reduction to an index means loss of informa-tion. Both of us have consistently tried throughout our careers toconvince scientists and others that indices can be misleadingand, if used at all, should not be used in isolation (e.g., Green,1979, pp 95–110; Chapman, 1996). We have had good companyin those attempts. A few examples: Hurlbert (1971) provided anearly critique of species diversity and of indices supposedly mea-suring it, in which he referred to ‘‘many semantic, conceptual,and technical problems’’. He suggested that ‘‘species diversity hasbecome a meaningless concept [and] that the term be abandoned’’.Eberhardt (1976) provided a critique of metrics in general, includ-ing diversity indices. He preferred model-based mathematical andstatistical analyses. Washington (1984) provides an excellent re-view of diversity, biotic and similarity indices in which he docu-ments how they are misused because they are often highlyspecialized to a particular type of water pollution (usually organicpollution), limited to specific geographic areas, and of limited eco-logical relevance.

More recent authors have also critiqued indices. Boyle et al.(1990) conducted a sensitivity analysis of nine diversity and sevensimilarity indices, found that some were insensitive to gross com-munity changes induced by pollution, and concluded that commu-nity level indices ‘‘may give very misleading biologicalinterpretations of the data they are intended to summarize’’. Izsákand Papp (2000) found that diversity indices were generally insen-sitive to both species differences and abundances. Thiebaut et al.(2002) noted that diversity indices do not necessarily provideany direct information on quality or degree of environmental deg-radation. Diaz et al. (2004) provide an excellent critical review ofmeasures of habit quality including biotic indices. Many differentdiversity indices have been proposed, including ‘‘information-based’’ ones (for summaries, see Dickman, 1968; Lloyd et al.,1968; Hurlbert, 1971; Hamilton, 1975). The various diversity indi-ces pretty much measure the same thing (i.e., are highly correlatedwhen calculated from real community data), so it doesn’t really

0025-326X/$ - see front matter � 2011 Elsevier Ltd. All rights reserved.doi:10.1016/j.marpolbul.2011.02.016

matter which one is used. For example, Auclair and Goff (1971) as-sessed diversity relations of 33 upland forest stands and demon-strated a high degree of correlation among 10 indices (eightbased on species abundances). One of us (RG) conducted a Princi-pal Components Analysis (PCA) on the data of Auclair and Goff(1971) and found that more than 75% of what is explained/pre-dicted by the indices was the same. So why not use the simplestdiversity measure, richness, when a diversity measure is calledfor? See also DeBenedictis (1973) regarding mathematically (notbiologically) driven correlations among diversity indices.

Many authors (e.g., Ricotta and Avena, 2003; Lamb et al., 2009;Dos Santos et al., 2011) criticize some indices and recommendusing others. We argue that this is a zero-sum exercise becausethe problems are common to all attempts to reduce communitystructure information to an index. We realize that regulatory ini-tiatives such as the Water Framework Directive in Europe encour-age the development of simplistic indices of water quality (Salaset al., 2006; Pinto et al., 2009), but they also caution in the stron-gest possible terms against believing that ecological complexitycan be adequately summarized by indices that reduce large massesof data to single numbers.

An index can be defined as a number derived from a formulathat summarizes some quantity of data. In environmental studiesindices are usually calculated from biological data (e.g., speciesabundances) and interpreted as responses to the environment.Depending on the purpose, ‘‘the environment’’ could mean theaverage natural environment (benign <=> harsh), a new or variableversus old and stable environment, or a human-impacted environ-ment. Indices reflecting natural community structure, such as spe-cies (or other taxonomic level) diversity indices have a longhistory. According to Pielou (1969), Fisher in the 1940s introducedthe idea of species diversity in connection with the log-series dis-tribution. Diversity indices have been used both to explain ‘‘undis-turbed’’ natural communities in relation to their environments andalso to infer degree of anthropogenic impact on communities (e.g.,Wilhm, 1972; Wilhm and Dorris, 1968). Here we focus on the lat-ter, but it is worth noting that there is a vast literature dealing withthe difficulties in inferring environmental causation from diversityindex values, even where the data are all from environments with-out any obvious anthropogenic disturbance.

For example, estuaries are harsh natural environments becauseof their low and fluctuating salinities and related osmotic prob-lems. Similarly, hypersaline environments such as endorheic pondsand lakes are harsh, but on a geological/evolutionary time scaleand a biogeographic spatial scale they are also new and variable– even ephemeral or intermittent. It has been argued that the estu-arine fauna are depauperate because estuarine environments aretransitional (between typical ocean salinities and fresh water)and short-lived, and there has not been enough time of stable

1378 Editorial / Marine Pollution Bulletin 62 (2011) 1377–1380

existence for the evolution of species adapted to those environ-ments. The same would be true of newly emerged volcanic islandsand temporary or fluctuating habitats such as the Dead Sea, GreatSalt Lake, or Australia’s Lake Ayre. So the point is: Which is it that islimiting species diversity – harsh environment or new and inter-mittent habitats/environments or both? The importance of changein this regard is generally underestimated. Treefalls in mature for-ests create ‘‘islands’’ of change and reversion to early succession.Even marine benthic communities at continental shelf depths(e.g., 100 m) respond to storm effects and re-start successional pro-cesses. When the fauna of the deep sea were first sampled theywere found to be surprisingly diverse, given the darkness, pressure,lack of photosynthesis, and low rates of organic material descend-ing from the upper layers. Biomass is low (except near volcanicvents) but diversity is high, as measured by richness (number oftaxa) or by any diversity index. A debate ensued which has generalimplications: what does control biotic diversity given that energy-poor deep sea environments support high diversity? The ‘‘Stability-Time Hypothesis’’ was proposed (Sanders, 1968, 1969; Dayton andHessler, 1972; Grassle and Sanders, 1973; Abele and Walters,1979), which essentially said that species diversity increasesasymptotically over time as species evolve and adapt to environ-ments. Disturbance in unstable environments sets back the processand reduces diversity. The greater faunal diversity of the Pacificthan the Atlantic Ocean has been attributed to the greater geolog-ical age of the Pacific.

How many aquatic pollution biologists who calculate diversityindices from species abundance data for monitoring purposes areaware of this literature? If they are not aware that different naturalenvironmental causes of given levels of diversity indices are diffi-cult to separate, then how can they be confident of correctly attrib-uting diversity index values to anthropogenic versus naturalcauses, when harsh and variable natural environments can yieldlow diversity values that are not pollution-related? One could ar-gue that sampling a particular habitat over time (the temporaldimension of a BACI design – Green, 1979) to detect changes indiversity would get around this problem, but this assumes thatany decreases in diversity are due to pollution impact rather thanto a natural environment change. There are many examples of thelatter being the case. For discussion of confounding of diversity andother biotic indices with natural spatial and temporal variation(see McGowan and Fraundorf, 1966; Pianka, 1966; Hilsenhoff,1998; Bergen et al., 2000; Hamilton, 2010). See Bergen et al.(2000) and Smith et al. (1999, 2001) for use of their Benthic Re-sponse Index (BRI) with a procedure for separating spatial gradi-ents of natural habitats (substrate, depth, latitude) from highversus low chemical exposure at a discharge.

Some who are aware of the spatial/temporal confoundingproblem propose using multimetrics, which include metrics fordifferent places or times such as seasons, thus compounding in-dex-confusion. To avoid the problem of ‘‘who knows exactlywhat diversity indices are responding to?’’, biotic indices havebeen derived to respond to pollution-induced changes in abun-dances of species that have been shown to be sensitive or resis-tant to specific contaminants (e.g., Hilsenhoff, 1987, 1998; Karr,1981, 1987, 1991; Kerans and Karr, 1994; Karr and Chu, 1999).A simple ratio of abundances of a number of sensitive speciesto a number of resistant species might exhibit the desired prop-erties, although such a ratio variable would have poor statisticalproperties (see discussion below). Such ‘‘purpose-derived’’ bioticindices transition into the indicator species concept (Smith et al.,1999; Bergen et al., 2000). Such ‘‘targeted’’ approaches are goodfor detection of particular pollution impacts selected a priori, butmay not respond interpretably if there is a different impact.Chessman and McEvoy (1998) propose constructing ‘‘a suite ofindices, each assembled using sensitivity numbers targeted to a

particular impact’’, to overcome this problem, a multimetric ap-proach (see below).

‘‘Multimetric’’ seems to have two meanings. Smith et al. (1999)describe one: combining ‘‘multiple measures of community re-sponse into a single index’’. But sometimes the meaning seems tobe to measure all sorts of things and report them all, hoping thateverything important has been included. Some multimetric refer-ences are: Paller and Specht (1997), Llanso et al. (2002), Whittieret al. (2007), and Stoddard et al. (2008). While the goals of ‘‘goingmultimetric’’ are often understandable, we can’t help reflectingthat they are usually the result of fundamental problems withthe index approach itself. The single biotic index is inadequatefor describing what is going on in the real world with its complexrelationships among biotic and environmental variables, spatialand temporal natural variation, and multiple anthropogenic stress-ors, so the response is to multiply indices or to use indices to cal-culate super-indices which are of course even less revealing aboutwhat is actually going on. One has to ask where the supposed sim-plicity of indices has gone and why it wouldn’t have been better toutilize other approaches in the first place.

Another problem with many indices is that they have bad sta-tistical properties, especially those which are ratios of variables(Sokal and Rohlf, 1973; Atchley et al., 1976; Green, 1979; Jackson,1997). For instance, many diversity indices are metrics that them-selves are fractions or percentages of taxa out of some total. Green(1986) described how ANCOVA with log–log regression can beused to analyze ratio variables, and presented worked examples.A number of authors (e.g., Heltshe and Forrester, 1983; Smithand Grassle, 1977) have discussed distributions of derived indicesused as response variables, and have proposed nonstandard proce-dures for analyzing them. However, one is still left with the sensethat it ought to be possible to analyze good data using standardclassical linear model normal distribution statistics, with simpletransformations.

Some feel that multivariate (MV) statistics are too difficult forstandard use. Norris (1995) thinks they are more sensitive forassessing perturbation than are metrics and indices, which he likes.The rest of the Reference Condition group (e.g., Reynoldson et al.,1997; Bailey et al., 2004) obviously agree, as do we. However thereis a common attitude that the implementation of MV approachesand assessment of their output are too complex to transmit easilyto managers (Smith et al., 1999 citing Gerritsen, 1995). Perhapswhat we need is better managers and better education of environ-mental scientists; in any case the Reference Condition approachwith MV statistical implementation has spread widely (mostlyoutside the US) with support and funding from government‘‘managers’’.

If indices must be calculated and presented then this should bedone together with other statistical methods that retain more ofthe information in the biological data set, e.g., an appropriate com-bination of univariate (UV) and MV statistical approaches (cf.Green, 1979; Chapman, 1996). For example, Reynoldson et al.(1997) found that precision and accuracy of MV methods wereconsistently higher than for multimetric assessment, but they rec-ommended that they be used together. Smith et al. (1999, 2001)used ordination to quantify a pollution gradient and then the toler-ance of each species was estimated from its distribution along thegradient. The BRI was calculated as the abundance-weighted aver-age pollution tolerance of species in a sample. Kilgour et al. (2004)compared seven indices with scores from three ordination axes.They found that the ordinations were more sensitive and con-cluded ‘‘we recommend that any suite of indices used for assessingbenthic communities should include these types of multivariatemetrics’’. This nicely illustrates how ordination can be used to findthe best linear additive model equivalent to an index, to produce a‘‘pollution score’’ for a sample. Griffith et al. (2002) used both

Editorial / Marine Pollution Bulletin 62 (2011) 1377–1380 1379

community metrics and a MV analysis to assess stream phyto-plankton assemblages in mineral-rich streams, and found thatthe two approaches were sensitive to different environmental fac-tors. Collier (2008) used eight metrics in a PCA (not a great idea wedon’t think) to develop a ‘‘Multivariate Condition Score’’, and com-pared it to Karr’s Index of Biotic Integrity. The Reference Conditionapproach can be implemented either with an index/metric ap-proach or a MV approach, or both.

Finally, there are other approaches, new ones that do not fit intoeither the index/metric category or the MV analysis category.Warwick and Clarke (1993, 1995, 1998) and Clarke and Warwick(1998a,b) have done pioneering work on new concepts related tocommunity response to pollution stress such as taxonomic dis-tinctness and structural redundancy.

In summary, avoid using indices because of information lossand the likelihood that their use will lead to misleading conclu-sions. If you absolutely must use indices for some non-scientificreason (hopefully not simply because your computer programcalculates them!), use them together with other statistical meth-ods that retain more of the information in the biological data set.Developing simplistic numbers simply to satisfy the leastknowledgeable scientists and managers is hardly the best wayto advance either scientific knowledge or managementdecision-making.

References

Abele, L.G., Walters, K., 1979. The stability-time hypothesis: reevaluation of thedata. American Naturalist 114, 559–568.

Atchley, W.R., Gaskins, C.T., Anderson, D., 1976. Statistical properties of ratios.1.Empirical results. Systematic Zoology 25, 137–148.

Auclair, A.N., Goff, F.G., 1971. Diversity relations of upland forests in the westernGreat Lakes area. American Naturalist 105, 499–528.

Bailey, R.C., Norris, R.H., Reynoldson, T.B., 2004. Bioassessment of FreshwaterEcosystems: Using the Reference Condition Approach. Kluwer, Boston, MA, USA.

Bergen, M., Cadien, D., Dalkey, A., Montagne, D.E., Smith, R.W., Stull, J.K., Velarde,R.G., Weisberg, S.B., 2000. Assessment of benthic faunal condition on themainland shelf of Southern California. Environmental Monitoring andAssessment 64, 421–434.

Boyle, T.P., Smillie, G.M., Anderson, J.C., Beeson, D.R., 1990. A sensitivity analysis ofnine diversity and seven similarity indices. Research Journal of the WaterPollution Control Federation 62, 749–762.

Chapman, P.M., 1996. Presentation and interpretation of Sediment Quality Triaddata. Ecotoxicology 5, 327–339.

Chessman, B.C., McEvoy, P.K., 1998. Towards diagnostic biotic indices for rivermacroinvertebrates. Hydrobiologia 364, 169–182.

Clarke, K.R., Warwick, R.M., 1998a. A taxonomic distinctness index and its statisticalproperties. Journal of Applied Ecology 35, 523–531.

Clarke, K.R., Warwick, R.M., 1998b. Quantifying structural redundancy in ecologicalcommunities. Oecologia 113, 278–298.

Collier, K.J., 2008. Linking multimetric and multivariate approaches to assess theecological condition of streams. Environmental Monitoring and Assessment157, 113–124.

Dayton, P.K., Hessler, R.R., 1972. Role of biological disturbance in maintainingdiversity in the deep sea. Deep Sea Research 19, 199–208.

DeBenedictis, P.A., 1973. Correlations between certain diversity indices. AmericanNaturalist 107, 295–302.

Diaz, R.J., Solan, M., Valente, R.M., 2004. A review of approaches for classifyingbenthic habitats and evaluating habitat quality. Journal of EnvironmentalManagement 73, 165–181.

Dickman, M., 1968. Some indices of diversity. Ecology 49, 1191–1193.Dos Santos, D.A., Molineri, C., Reynaga, M.C., Basualdo, C., 2011. Which index is best

to assess stream health? Ecological Indicators 11, 582–589.Eberhardt, L.L., 1976. Quantitative ecology and impact assessment. Journal of

Environmental Management 4, 27–70.Gerritsen, J., 1995. Additive biological indices for resource management. Journal of

the North American Benthological Society 14, 451–457.Grassle, J.F., Sanders, H.L., 1973. Life histories and the role of disturbance. Deep Sea

Research 20, 643–659.Green, R.H., 1979. Sampling Design and Statistical Methods for Environmental

Biologists. Wiley, New York, NY, USA.Green, R.H., 1986. Some applications of linear models for analysis of contaminants

in aquatic biota. In: El-Shaarawi, A.H., Kwiatowski, R.E. (Eds.), Statistical Aspectsof Water Quality Monitoring. Elsevier, New York, NY, USA, pp. 231–245.

Griffith, M.B., Hill, B.H., Herlihy, A.T., Kaufmann, P.R., 2002. Multivariate analysis ofperiphyton assemblages in relation to environmental gradients in ColoradoRocky Mountain streams. Journal of Phycology 38, 83–95.

Hamilton, A.T., 2010. Vulnerability of biological metrics and multimetric indices toeffects of climate change. Journal of the North American Benthological Society29, 1379–1396.

Hamilton, M.A., 1975. Indices of diversity and redundancy. Journal of the WaterPollution Control Federation 47, 630–632.

Heltshe, J.F., Forrester, N.E., 1983. Estimating species richness using the jackknifeprocedure. Biometrics 39, 1–11.

Hilsenhoff, W.L., 1987. An improved biotic index of organic stream pollution. GreatLakes Entomology 20, 31–40.

Hilsenhoff, W.L., 1998. A modification of the biotic index of organic stream pollutionto remedy problems and permit its use throughout the year. Great LakesEntomology 31, 1–12.

Hurlbert, S.H., 1971. The nonconcept of species diversity: a critique and alternativeparameters. Ecology 52, 577–586.

Izsák, J., Papp, L., 2000. A link between ecological diversity indices and measures ofbiodiversity. Ecological Modeling 130, 151–156.

Jackson, D.A., 1997. Compositional data in community ecology: the paradigm orperil of proportions. Ecology 78, 929–940.

Karr, J.R., 1981. Assessment of biotic integrity using fish communities. Fisheries 6,21–27.

Karr, J.R., 1987. Biological monitoring and environmental assessment: a conceptualframework. Environmental Management 11, 249–256.

Karr, J.R., 1991. Biological integrity - long-neglected aspect of water-resourcemanagement. Ecological Applications 1, 66–84.

Karr, J.R., Chu, E.W., 1999. Restoring Life in Running Waters: Better BiologicalMonitoring. Island Press, Washington, DC, USA.

Kerans, B.L., Karr, J.R., 1994. A benthic index of biotic integrity (B-IBI) for rivers ofthe Tennessee Valley. Ecological Applications 4, 768–785.

Kilgour, B.W., Somers, K.M., Barton, D.R., 2004. A comparison of the sensitivity ofstream benthic community indices to effects associated with mines, pulp andpaper mills, and urbanization. Environmental Toxicology and Chemistry 23,212–221.

Lamb, E.G., Bayne, E., Holloway, G., Schieck, J., Boutin, S., Herbers, J., Haughland, D.L.,2009. Indices for monitoring biodiversity change: are some more effective thanothers? Ecological Indicators 9, 432–444.

Llanso, R.J., Scott, L.C., Hyland, J.L., Dauer, D.M., Russell, D.E., Kutz, F.W., 2002. Anestuarine benthic index of biotic integrity for the Mid-Atlantic region of theUnited States. II. Index development. Estuaries 25, 1231–1242.

Lloyd, M., Zar, J.H., Karr, J.R., 1968. On the calculation of information-theoreticalmeasures of diversity. American Midland Naturalist 79, 257–272.

McGowan, J.A., Fraundorf, V.J., 1966. Relationship between size of net used andestimates of zooplankton diversity. Limnology and Oceanography 11, 456–461.

Norris, R.H., 1995. Biological monitoring: the dilemma of data analysis. Journal ofthe North American Benthological Society 14, 440–450.

Paller, M.H., Specht, W.L., 1997. A multimetric index using macroinvertebrate datacollected with artificial substrates. Journal of Freshwater Ecology 12, 367–378.

Pianka, E.R., 1966. Latitudinal gradients in species diversity – a review of concepts.American Naturalist 100, 33–40.

Pielou, E.C., 1969. An Introduction to Mathematical Ecology. Wiley, New York, NY,USA.

Pinto, R., Patricio, J., Baeta, A., Fath, B.D., Neto, J.M., Marques, J.C., 2009. Review andevaluation of estuarine biotic indices to assess benthic condition. EcologicalIndicators 9, 1–25.

Reynoldson, T.B., Norris, R.H., Resh, V.H., Day, K.E., Rosenberg, D.M., 1997. Thereference condition: a comparison of multimetric and multivariate approachesto assess water-quality impairment using benthic macroinvertebrates. Journalof the North American Benthological Society 16, 833–852.

Ricotta, C., Avena, G., 2003. On the relationship between Pielou’s evenness andlandscape dominance within the context of Hill’s diversity profiles. EcologicalIndicators 2, 361–365.

Salas, F., Marcos, C., Neto, J.M., Patricio, J., Perez-Ruzafa, A., Marques, J.C., 2006. User-friendly guide for using benthic ecological indicators in coastal and marinequality assessment. Ocean and Coastal Management 49, 308–331.

Sanders, H.L., 1968. Marine benthic diversity: a comparative study. AmericanNaturalist 102, 243–282.

Sanders, H.L., 1969. Benthic marine diversity and the stability-time hypothesis.In: Woodwell, G.M., Smith, H.H. (Eds.), Diversity and Stability in EcologicalSystems. Brookhaven Symposium in Biology No. 2. Brookhaven, New York,NY, USA.

Smith, R.W., Bergen, M., Weisberg, S.B., Cadien, D., Dalkey, A., Montagne, D., Stull,J.K., Velarde, R.G., 1999. Benthic Response Index for assessing infaunalcommunities on the mainland shelf of Southern California. In: Weisberg, S.B.,Elmore, D. (Eds.), Southern California Coastal Water Research Project 1997–1998 Annual Report. SCCWRP, Westminster, CA, USA.

Smith, R.W., Grassle, J.F., 1977. Sampling properties of a family of diversitymeasures. Biometrics 33, 283–292.

Smith, R.W., Bergen, M., Weisberg, S.B., Cadien, D., Dalkey, A., Montagne, D., Stull,J.K., Velarde, R.G., 2001. Benthic Response Index for assessing infaunalcommunities on the Southern California mainland shelf. EcologicalApplications 11, 1073–1087.

Sokal, R.R., Rohlf, F.J., 1973. Introduction to Biostatistics. Freeman, San Francisco, CA,USA.

Stoddard, J.L., Herlihy, A.T., Peck, D.V., Hughes, R.M., Whittier, T.R., Tarquinio, E.,2008. A process for creating multimetric indices for large-scale aquatic surveys.Journal of the North American Benthological Society 27, 878–891.

1380 Editorial / Marine Pollution Bulletin 62 (2011) 1377–1380

Thiebaut, G., Guérold, F., Muller, S., 2002. Are trophic and diversity indices based onmacrophyte communities pertinent tools to monitor water quality? WaterResearch 36, 3602–3610.

Warwick, R.M., Clarke, K.R., 1993. Increased variability as a symptom of stress inmarine communities. Journal of Experimental Marine Biology and Ecology 172,215–226.

Warwick, R.M., Clarke, K.R., 1995. New ‘biodiversity’ measures reveal a decrease intaxonomic distinctness with increasing stress. Marine Ecology Progress Series129, 301–305.

Warwick, R.M., Clarke, K.R., 1998. Taxonomic distinctness and environmentalassessment. Journal of Applied Ecology 35, 532–543.

Washington, H.G., 1984. Diversity, biotic and similarity indices – a review withspecial relevance to aquatic ecosystems. Water Research 18, 653–694.

Whittier, T.R., Hughes, R.M., Stoddard, J.L., Lomnicky, G.A., Peck, D.V., Herlihy, A.T.,2007. A structured approach for developing indices of biotic integrity: threeexamples from streams and rivers in the western USA. Transactions of theAmerican Fisheries Society 136, 718–735.

Wilhm, J., 1972. Graphic and mathematical analyses of biotic communities inpolluted streams. Annual Review of Entomology 17, 223–252.

Wilhm, J., Dorris, T.C., 1968. Biological parameters for water quality criteria.BioScience 18, 477–481.

Roger GreenDepartment of Biology, University of Western Ontario,

London, ON, Canada N6A 5B7E-mail address: rgreen@uwo.ca

Peter M. ChapmanGolder Associates Ltd., 500 – 4260 Still Creek Drive,

Burnaby, BC, Canada V5C 6C6E-mail addresses: pmchapman@golder.com