evaluating data quality collected by volunteers for ﬁrst ... · teers due to safety limitations...

Nat Hazards Earth Syst Sci 14 2681ndash2698 2014wwwnat-hazards-earth-syst-scinet1426812014doi105194nhess-14-2681-2014copy Author(s) 2014 CC Attribution 30 License

Evaluating data quality collected by volunteers for first-levelinspection of hydraulic structures in mountain catchments

V J Cortes Arevalo12 M Charriegravere 1 G Bossi2 S Frigerio2 L Schenato2 T Bogaard1 C Bianchizza3 A Pasuto2and S Sterlacchini4

1Delft University of Technology Delft the Netherlands2Italian National Research Council Institute for Geo-hydrological Protection CNR-IRPI Padova Italy3Institute of International Sociology Gorizia ISIG Gorizia Italy4Italian National Research Council Institute for the Dynamic of Environmental Processes CNR-IDPA Milan Italy

Correspondence toV J Cortes Arevalo (juliettecortesirpicnrit)

Received 15 April 2014 ndash Published in Nat Hazards Earth Syst Sci Discuss 22 May 2014Revised 30 July 2014 ndash Accepted 31 July 2014 ndash Published 6 October 2014

Abstract Volunteers have been trained to perform first-levelinspections of hydraulic structures within campaigns pro-moted by civil protection of Friuli Venezia Giulia (Italy)Two inspection forms and a learning session were prepared tostandardize data collection on the functional status of bridgesand check dams In all 11 technicians and 25 volunteers in-spected a maximum of six structures in Pontebba a moun-tain community within the Fella Basin Volunteers includedcivil-protection volunteers geosciences and social sciencesstudents Some participants carried out the inspection with-out attending the learning session Thus we used the modeof technicians in the learning group to distinguish accuracylevels between volunteers and technicians Data quality wasassessed by their accuracy precision and completeness Weassigned ordinal scores to the rating scales in order to getan indication of the structure status We also considered per-formance and feedback of participants to identify correc-tive actions in survey procedures Results showed that vol-unteers could perform comparably to technicians but onlywith a given range in precision However a completenessratio (question parameter) was still needed any time vol-unteers used unspecified options Then volunteersrsquo ratingscould be considered as preliminary assessments without re-placing other procedures Future research should consideradvantages of mobile applications for data-collection meth-ods

1 Introduction

There is an increasing interest in the use of citizen-based ap-proaches to better understand the environment and hazard-related processes To that end there are different data-collection approaches according to the citizensrsquo skills andtime of involvement eg crowdsourcing (Hudson-Smithet al 2008) volunteered geographic information (Good-child 2007) or facilitated-volunteered geographic informa-tion (Seeger 2008) Moreover scientists have increasinglyconsidered management approaches based upon the broaderconcept of citizen science (Bonney et al 2009) Therebyvolunteers are enlisted and trained according to survey andmanagement needs (Devictor et al 2010)

In disaster risk management citizen science is linkedto European and worldwide directives such as the Hyogoframework (European Commission 2007 United Nations2005) Such directives promote citizen involvement to builda culture of resilience before during and after a disas-ter strikes (European Commission 2012) Therefore mod-ern approaches for emergency management promote ex-change of information between local authorities and volun-teer groups to support preparedness and preventive actions(Enders 2001)

Hydro-meteorological events in mountain areas are of-ten caused by multiple and sudden onset floods and debrisflows Traditionally hazard mitigation in the European Alpsis mainly organized by implementing structural measuresHowever the increasing frequency and influence of flow and

Published by Copernicus Publications on behalf of the European Geosciences Union

2682 V J Cortes Arevalo et al Quality of volunteersrsquo hydraulic-structure inspections

sediment processes also affect the functional status of hy-draulic structures and vice versa (Holub and Huumlbl 2008)

The impact (ie damage) is evident to structures fordebris-flow control such as check dams Evidence is alsofound in the potential aggravation of flood hazard at the lo-cation of bridges and culverts due to blocking material suchas debris large wood and other residues (Mazzorana et al2010) Stability of protection works is often threatened bythe erosion level at stream banks

The need to enhance data-collection approaches to supportrisk-management strategies is widely acknowledged (egMolinari et al 2014) Besides situations where financial andhuman resources are limited scientific monitoring may besubject to additional complexity under dynamic environmen-tal conditions or remote settings (de Jong 2013)

Moreover frequent inspection of hydraulic structures isespecially important in mountain basins Therefore oppor-tunities in promoting citizen science projects stem from theincreasing frequency timeliness and coverage of surveillanceactivities (Flanaging and Metzger 2008) To be useful sur-vey procedures should be tested and adapted according toquality requirements of decision-makers (Bordogna et al2014 EPA 1997 Goodchild and Li 2012 Gouveia and Fon-seca 2008)

Experiences of citizen science include data collection re-garding water quality and biological aspects (Engel andVoshell Jr 2002 Fore et al 2001 Nicholson et al 2002)forestry and ecosystem rehabilitation (Brandon et al 2003Gollan et al 2012) biodiversity (Snaumlll 2011) stream mon-itoring (Bjorkland et al 2001 Yetman 2002) and hy-drological processes (Cifelli et al 2005 Rinderer et al2012) Despite the variety of projects limited researchhas been devoted to evaluating the quality of citizen-baseddata (Danielsen et al 2005) Furthermore scientists anddecision-makers have a general lack of confidence due to thelimited accuracy non-comparability and incompleteness ofcitizen-collected data (Riesch and Potter 2014 Conrad andHilchey 2011)

In spite of the challenges for citizen involvement preci-sion and completeness of largely collected data depends onthe exhaustiveness of the inspection procedures (Gallowayet al 2006) Therefore training activities are often requiredbefore starting the inspection campaigns However the ex-tension of these training sessions should consider availabletime number and type of participants (Tweddle et al 2012)To that endJordan et al(2011) suggested that identifica-tion of technical data should be restricted while more gen-eral indicators can still be accurately obtained Those indica-tors may be quantitative and qualitative aspects that are eas-ily recognizable from visual inspections (Gommerman andMonroe 2012 Gouveia et al 2004) Then qualitative fieldmethods are generally based on rating scales to report in-spected conditions

This study considers regular inspections with citizen-volunteer groups that are promoted by civil protection andlocal authorities of Friuli Venezia Giulia (FVG) Italy We in-volved 11 technicians and 25 volunteers on a data-collectionexercise Participants were invited according to their lo-cation Thus 15 out of 25 volunteers were members ofcivil-protection groups in neighboring municipalities In all10 university students were also volunteers within a supple-mentary academic activity

In this paper we evaluate data quality on preliminary in-spections of bridges and check dams Therefore we addressthe following research questions (1) how well were partici-pants able to report on the functional status by distinguishingbetween available rating classes (2) How effectively weredata collected by volunteers compared to those collected bytechnicians (3) How can survey procedures be improvedTo that end Sect2 describes the methodology for data col-lection In Sect3 we evaluate data quality by their accuracyprecision and completeness Finally we highlight in the dis-cussionrsquos and conclusionrsquos (Sects4 and5) key points for thepractical use of citizen-based data

2 Methods

In the first step of the methodology we defined target groupsThe participantsrsquo groups comprised of volunteers and tech-nicians to evaluate the data quality In the second step twoinspection forms were designed for bridges and check damsThe forms were created to carry out inspections with trainedvolunteers

Despite available procedures for technicians the volun-teersrsquo involvement demands more structured and simplerforms to inspect the functional status Similar toYetman(2002) we used rating scales to standardize collected data indistinguishing minor problems from more serious concernsThe rating scales included visual schemes to guide inspec-tors In addition the latter were required to take a photo tosupport their choices

Finally we organized a data-collection exercise to carryout first-level inspection of six structures hereafter referredto as ldquoinspection testsrdquo There was 1 day for the training ses-sion and 1 day for the inspection tests Participants were di-vided between control and learning groups to identify poten-tial improvements in survey procedures

21 Participantsrsquo groups

Citizens were involved in the form of civil-protection volun-teers due to safety limitations and accessibility to hydraulicstructures in mountain catchments Citizens enrolled as civil-protection volunteers traditionally received formative infor-mative and safety procedures while specialized training is se-lectively provided (Protezione Civile FVG 2009)

Nat Hazards Earth Syst Sci 14 2681ndash2698 2014 wwwnat-hazards-earth-syst-scinet1426812014

V J Cortes Arevalo et al Quality of volunteersrsquo hydraulic-structure inspections 2683

Table 1Participantsrsquo distribution between learning groups (LGs) and control groups (CGs)

Participantrsquos number 2-day LG 1-day CG

Civil protection volunteers 15 8 from Pontebba 1 from Pontebba (Date 1)2 from Chiusaforte 1 from Dogna (Date 3)2 from Dogna 1 from Malborghetto (Date 4)

Geo-sciences students 8 4lowast 4Social science students 2 2 No participants

Volunteers (Vs) 25 18 learning (of which 14 testing) 7 testing

Civil protection (CP) 11 3 and 1lowast from CP 1 from CP (Date 3)Forestry service (FS) 1 from FS 1 from FS (Date 1)Geological survey (GS) 1 from GS 3 from GS (Date 2)

Technicians (Ts) 11 6 learning (of which 5 testing) 5 testing

Participants 36 24 learning (of which 19 testing) 12 testing

lowast LG participants that attended only 1-day (pre-test and test 1 for bridges and check dams)

In addition we widened the range of participants withstudents to account for assumed differences in preliminaryknowledge to fill the form Then volunteers included geol-ogy students and students from a masterrsquos course on coop-eration both from the University of Trieste The technicianswere employees from regional services with competences forthe inspection of hydraulic structures in the mountain com-munity

Volunteers (Vs) and technicians (Ts) joined the activity ac-cording to their time-availability (Table1) The control group(CG) carried out the inspection tests without attending thelearning session Most citizen-volunteers were present in thelearning group (LG) as they are the target group of cam-paigns promoted by civil protection Students of geoscienceswere only available for the inspection tests during the firstday Their involvement was important to facilitate knowledgeexchange during outdoor learning Then they were equallydivided into a volunteersrsquo learning group (VLG) and a vol-unteersrsquo control group (VCG)

During registration to the data-collection exercise partic-ipants were asked to fill a questionnaire to characterize par-ticipantsrsquo groups (TableA1) Gathered information includeddemographics such as age gender and level of education aswell as period of residence in the Fella Basin and FVG re-gion In addition we measured the experience with hydro-meteorological hazards (ie floods debris flows and land-slides) The questionnaire also included 20 questions to as-sess a prior knowledge of participants of debris-flow phe-nomena functionality of check dams and culverts as well asemergency security guidelines

22 Design of the inspection forms

Table2 summarizes the formsrsquo layouts divided by sectionsWe defined the latter with four risk managers of civil protec-tion the geological survey and the forestry service of FVG

Section I identifies the inspector Section II comprises simpli-fied information on location type use and presence of con-nected structures if available Section III accounts for thelevel of accessibility presence of stream water occurrenceof rainfall and snow Thus section III becomes relevant forcomparing between campaigns carried out at different timeperiods Thereafter section IV of the form refers to the func-tional status of the inspected structure Functional status isthe susceptibility or physical conditions of the structure thatmay affect the function type for which it was designed orbuilt (Uzielli et al 2008) Furthermore the functional statusis inspected by looking at three parameters according to thestructure type

Parameters in section IV are comprised of a maximum offour questions For example questions for check dams in pa-rameter A are (1) is the stream flow passing where it shouldbe (2) What is the status of the check dam (3) How visibleis the basis of the structure (4) Is there any protection forscouring at the downstream bottom of the check dam Ques-tions rating options and visual schemes were defined ac-cording to inspection procedures for technicians Those weremainly for check dams (Jakob and Hungr 2005 ProvinziaAutonoma di Bolzano 2006 von Maravic 2010 Provinceof British Columbia 2000) and bridges (Burke Engineering1999 Ohio Department of Transportation 2010 ServizioForestale FVG 2002) The inspection forms adopted in thisstudy are available as the Supplement

For the case of bridges parameter A focuses on the open-ing for the water flow and erosion of the pillar or abutmentsParameter B assesses levels of lateral obstruction either atthe structure location or at the stream channel Thereforequestions for these parameters were aimed at identifyingmorphological changes immediately upstream downstreamand at the structure location Such changes relate to eitherlocal erosion at protection structures and abutments depo-sition phenomena that is somehow perennial with presence

wwwnat-hazards-earth-syst-scinet1426812014 Nat Hazards Earth Syst Sci 14 2681ndash2698 2014


of vegetation or clogging of critical flow sections Finallyit accounts for additional elements such as pipes when theyreduce the stream cross section

We also included a question referring to the maximum freeheight of the structure However it was in the end not con-sidered due to safety limitations for citizen volunteers whenaccessing the stream channel Limitations are based on thedynamic distribution of deposits and eroding surfaces alongsteep mountain channels (Remaicirctre et al 2005) Thereforevolunteers should follow safety procedures according to theenvironmental and meteorological conditions during the in-spection period

For check dams the focus of parameter A is on the statusof the structure itself and downstream scouring Parameter Bdistinguishes between consolidation and open check damsThen upstream obstruction is limited to the open check damtype That distinction is due to the relevance of open checkdams for retention of sediments if there is a retention basinconnected to the structure Therefore we included a ldquoDoesnot applyrdquo option for inspecting consolidation check dams

In contrast parameter C addresses the same questions forbridges and check dams It refers to the worst condition whilelooking at the presence of protection works and erosion levelat the stream banks Then we established a control distanceof 20 m upstream and downstream of the structure This dis-tance was defined to reduce variability of assessments dur-ing the inspection The 20 m allow inspectors to observe andto take pictures even if accessibility to the structure is re-stricted Section V of the form reports the critical infrastruc-ture within the same control distance Finally section VI dis-tinguishes required actions to follow up the inspection basedon the options provided in the form

Data-quality evaluation focuses on sections IV and VI ofthe inspection form Table3 summarized the rating scaleswe used When the question itself did not specify the loca-tion to report a multiple choice was included by specifyingthe problemrsquos location right left or in correspondence withthe structure In addition all questions had alternative op-tions to report unspecified answers such as ldquoI donrsquot knowrdquoand ldquoCould not be answeredrdquo The latter represents condi-tions at the structure location (eg water level) that did notallow inspectors to provide an assessment

In addition we assigned ordinal scores to the rating classesto get an indication on the functional status For the data-quality evaluation we aggregated scores according to thegiven range in precision while generalizing the rating scales

23 Data-collection exercise

The data-collection exercise was carried out in the munici-pality of Pontebba (FVG Italy) within the mountain com-munity in the Fella Basin (Fig1a) The only settlement withover 4000 inhabitants is Tarvisio bordered by Austria andSlovenia The Fella catchment has an area of 700 km2 witha mean altitude of 1140 m asl It consists mostly of lime-

stone and it is characterized by steep slopes and high tectonicgrade The area is prone to landslides flash floods and debrisflows

The latest severe alluvial event was on 29 August 2003The total rainfall amount of the event was equal to 3896 mmDetected intensities were particularly strong for values corre-sponding to 3 and 6 h (Borga et al 2007) The event causedsevere damage created gullies and expanded existing riverbeds The most affected villages were Ugovizza ValbrunaMalborgehtto and Pontebba (Calligaris and Zini 2012)

After the 2003 event technical services updated the inven-tory of debris flows Civil protection realized several mitiga-tion measures in the affected areas The basin authorities pro-duced an updated version of the hazard maps PAI-FELLA(ADBVE 2012) Thus 22 of total check dams and 50 total of bridges within Fella Basin are accounted at the dif-ferent hazardous areas defined in the PAI upstream of thePontebba location That corresponds to 230 and 115 struc-tures respectively

Civil protection selected the structures for the inspectiontests (Fig1b) The complexity of the inspection tests dif-fered according to the functional status of the structuresThen structures for the inspection tests accounted for a rangefrom minimal to serious concerns Structures also includedconnected elements such as retention basins and secondarycheck dams for scouring protection

Finally Table4 describes the organization of the exer-cise divided by sessions and inspection tests The registra-tion questionnaire was open until the exercise day whichtook place on May 2013 See the website of the activity1 formore details on the questionnaire and training material

After a common introductory session participants in-spected the same structures according to the CG or LG pro-gram Every test took 15 min on average which was actuallyfaster than expected First all participants carried out an in-door pretest by looking at a poster Then the CG initiatedinspections directly in the field without attending the learn-ing session The CG program was on different dates based onthe participantsrsquo availability

Instead the LG continued their first-day with the learningsession The learning had an indoor program followed by anoutdoor session that included the structures of the pretest Inthe outdoor session the LG was divided into teams with rep-resentatives of each participantrsquos group (one technician civil-protection volunteers and one student) At first each partic-ipant compiled the test 1 for check dams and bridges Thenthey filled it out by teams for knowledge exchange Seniortechnicians clarified further aspects if needed On the sec-ond day the LG continued with test 2 and 3 divided in twogroups to reach the structure location At the end all partici-pants provided feedback and submitted the pictures they tookduring the inspections if any

1httpshoratiusirpipdcnritchanges-fellachanges-fellaindexphprelazione-dell-attivita



Table 2Formrsquos layout for bridges and check dams

Sections Aim of each section

I Inspectorrsquos name and period of the inspection Identify person responsible for form compilingIdentify time and inspection period based on rainfall conditionsduring last 24 h if known

II Structure and function type Precompiled with data available from regional databases of hydraulic structures

III Inspection conditions Distinguish conditions of regular inspections from those to intensify surveillance

IV Functional status Distinguish among the following possible actionsA Condition of the structure No action is requiredB Level of obstruction at the structure Requires routine cleaning of blockages by hand

with a maximum group of 10 volunteersC Presence of protection works and Requires cleaning with support of equipment

erosion level at the stream bank Second level inspection actions other than cleaning are required

V Presence of anthropic elements Refer critical infrastructure next to the structure

VI Synthesis of the inspection Provide general recommendation from available options

Table 3Rating scales used in the inspection form

Rating classes Criteria Scorersquos meaning Generalized classes Questions(Scores) (Precision range)

Five classes Minimum to 1 Best condition Three classes Bridges B1 amp B2(1 2 3 4 and 5) maximum 5 Worst condition (1ndash2 3 and 4ndash5) Check dams A2 A3 B1 amp B3

concerns Both C21 2 3 amp 4

Three classes Differences 1 Best condition Three classesCheck dams A4 amp B1(1 3 and 5) are not for 3 Medium condition (None)

five classes 5 Worst condition

Three classes Relevance 1 Total absence Two classes Bridges A1lowast

(1 4 and 5) to present 4 Presence (1 and 4ndash5) Check dams A1lowast

aspects 5 Presence beyond reference aspect Both C11 2 3 amp 4

Two classes Yesno 1 Absence Two classesBridges A2lowast A3 amp B3lowast

(1 and 5) 5 Presence (None)

Synthesis Management options for Two classesSection VI of the form

(1 2 4 and 5) synthesis of the inspection (1ndash2 and 4ndash5)

lowast Includes multiple choice to specify the location of the problem

3 Evaluation on the quality of collected databy volunteers

Tables5 and6 summarize results according to the componentquestions per parameter Mean ordinal scores(X) and stan-dard deviations (SD) were calculated from the ratings thatparticipants reported in test 1 2 and 3

Then we evaluated how effectively data were collected byVs as compared to that collected by Ts Figures2ndash6 summa-rize results according to the pretest tests 1 2 and 3 Distinc-tion was done between Vs and Ts within participants of thelearning and control groups For that purpose a frequencyanalysis was applied to the ordinal scores that Table3 de-

fines This consideration was based upon the relatively lowsample size and difference in number between the groupsWe referred to the mode score for the data evaluation as itrepresents the class with the highest frequency In additionwe used the following criteria to assess the quality of col-lected data (EPA 1997 p 19ndash20)

ndash Accuracy is a ldquodegree of agreement between the datacollected and the true value on the condition being mea-suredrdquo Then we referred to as ldquotrue valuerdquo as the modescore for Ts in the learning group Figures2ndash6 aggre-gate the relative frequencies in four frequency classeswith reference to the true value equal to or larger than



Figure 1 Study area(a) overview of the study area and Fella Basin(b) Map indicating the location of the structures for the inspection tests

Table 4Description of the data-collection exercise

Session Inspection test Description of activities

(Duration) (Structure) 2-day LG 1-day CG

(1) Registration Filling the registration questionnaire(1 month)

(2) Introduction Pretest with a poster Scope of the activity(1 h) (Bridge 1 and check dam 1) Safety advice and recommendations

for form compilingPre-test to establish the initialunderstanding of participants to fill the form

(3a) Indoors learning Presentations of 45 min separated by breaks in-between Participants divided in four groups(3 h)

(3b) Outdoor learning Test 1 Participants divided in two teams Each group made the test in(3 h) (Bridge 1 and check dam 1) guided by a senior technician one of three optional dates

Individual form compiled for two structures Individual form compiledTeam divided in five sub teams of five participants for six structuresGroup form compiled per sub team

(4) Testing Test 2 and 3 Participants divided in two teams to minimize interaction(3 h) (Bridge 2 and 3) Individual forms compiled for four structures

(Check dam 2 and 3)

(5) Feedback Feedback form (15 min) and during learning sessions

90 70ndash90 and 50ndash70 and smaller than 50 Wechose this aggregation to distinguish different accuracylevels for each group In addition we assumed agree-ment among group members when a question had a rel-ative frequency of at least 70 Then the overall agree-

ment per parameter was calculated by the ratio ques-tion parameter ie the number of questions with fre-quencies of at least 70 between total questions perparameter



Figure 2 Relative frequencies for Vs and Ts of learning groups (LGs) and control groups (CGs) Parameters A and B in inspection tests forbridges

ndash Precision ldquorefers to how well data collected are able toreproduce the result on the same grouprdquo For all partic-ipants we represented precision by using the standarddeviation (SD) in Tables5 and6 Instead in Figs2ndash6 we compare each group while looking at the modescores and the mode-off by one level The mode-offis a range in precision given by generalizing the ex-treme scores of the rating classes For example accord-ing to Table3 we generalize rating scales from five tothree classes by grouping very low to low concernsmedium concerns and high to very high concerns Thoseare ordinal scores 1 and 2 on one side scores 4 and 5on the other Mode-off by one level in Figs2ndash6 onlydistinguished questions where the scale generalizationbrought forth increments to the relative frequencies

ndash Completeness is the ldquomeasure of the amount of validdata actually obtained vs the amount expected to be ob-tainedrdquo In Tables5 and 6 completeness is evaluatedby the amount of answers obtained between the rat-ing scales as compared to the selection of unspecifiedanswers In Figs2ndash6 we evaluated completeness bydistinguishing questions with relative frequencies largerthan 14 in the options ldquoI donrsquot knowrdquo ldquoCould not beansweredrdquo and ldquoNo answerrdquo We chose a threshold of14 to highlight questions with the lower complete-

ness It corresponds approximately to an absolute fre-quency of one participant in the control group or twoparticipants in the learning group

Other criteria such as comparability and representativenesswere only considered in designing the form ldquoComparabil-ity represents how well data from one form can be com-pared to data from another Representativeness is the degreeto which collected data actually represent the structure be-ing inspectedrdquo (EPA 1997 p 19ndash20) Then we referred tocomparability by using a standard form for bridges and checkdams For representativeness we required a photo recordfrom the inspector to support their choices and to provide ad-ditional information for the later examination of inspections

Finally we used comments provided by participants dur-ing the sessions and the comments provided in the feedbackform to define corrective actions (Table7) Following sub-parts present results according to specific aspects for bridgesand check dams then into common aspects for both struc-tures

31 Functional status of bridges for A and B parameters

Table5 shows that A and B parameters in test 1 have meanscores between 1 and 2 in the functional status Lower ordi-nal scores represent the best condition for inspected aspects



Figure 3 Photo record of the structures inspected(a) upstreamview of bridge 1 pretest and test 1(b) Upstream view of bridge2 test 2(c) Upstream view of bridge 3 test 3(d) check dam 1pretest and test 1(e) Opening of check dam 2 test 2(f) Upstreamto downstream view of check dam 3 test 3

(Fig 3a) Despite Ts in the control group Fig2 presentsoverall agreement near to one for parameter A That is theratio question parameter for parameter A and test 1 For pa-rameter B overall agreement was reached only in the mode-off by one level That represents lower precision in the B rat-ings indistinctly of the groups

However performance in test 1 contrasts with the one intest 2 Inspection complexity of bridge 2 was higher due tostream water flowing along the structurersquos pillars and abut-ments (Fig3b) For parameter A Table5 highlights higherfrequency of unspecified answers and ordinal scores withstandard deviations larger than 1 For Parameter A Fig2shows accuracy levels below the relative frequency of 70 Consequently there is disagreement in the mode score be-tween Vs and Ts indistinctly of the groups The presence oferosion in test 2 ie question A1 was mostly rated by par-ticipants as ldquoCould not be answeredrdquo ldquoNo answerrdquo or ldquoNoerosionrdquo Moreover those who reported erosion in the pil-lar and abutment did not distinguish among erosion presencewith or without the stream water along the basis

For parameter A in test 3 Fig2 shows better perfor-mance for the TLG and VLG as compared to the TCG andVCG The difference in performance could represent someinfluence of the learning session However it also denotesthe need for adjusting questions to avoid misunderstandings

That is the case of question A3 which should explicitly ad-dress the status of protection works for downstream scouringin bridges (Fig3c) For parameter B Ts and Vs only reachedaccuracy levels above 70 when looking at the mode-offby one level indistinctly of the test However question B3(presence of islands with shrubs or man-made structures thatreduce the opening for the flow) had the lowest precision inTable5 and Fig2 Then question B3 should be split for bet-ter distinguishing presence of islands with vegetation fromman-made obstructions

32 Functional status of check dams for parameters Aand B

Table6 highlights the questions with more serious concernsand standard deviations above 1 Despite the functional sta-tus the presence of connected elements to the structures con-tributed to larger standard deviations Thus complexity inthe inspection was higher due to the presence of a secondarystructure in check dam 1 and a retention basin in check dam2 (Fig3d and e)

Figure 4 shows the lower accuracy levels and overallagreement ratio for parameter A in test 3 Then question A1was the least accurate for Vs in test 2 and 3 Those resultsmay be explained on the rating scale we used (see Supple-ment for check dams) For question A1 rating classes didnot distinguish slight deviations from the strong ones Thenmedium concerns were not explicitly within the available op-tions

In addition Fig4 shows higher frequencies of unspeci-fied answers for questions A2 and A3 in test 1 and 2 In test1 unspecified answers were due to the water level at the ba-sis of the structure ldquoCould not be answeredrdquo was even thepreferred option for the TLG In test 2 visibility at the basisof the structure was limited due to the sediment accumula-tion Finally question A4 denoted higher frequencies of un-specified answers for all structures (Table6 and Fig4) De-scription of question A4 should be reviewed to avoid misun-derstanding with question A2 That is the case of connectedstructures for protection of downstream scouring Classes toreport in question A4 should be extended to consider all pos-sible functional conditions

For parameter B questions B1 and B2 have the lowestcompleteness in pretest and test 1 Those questions were notrelevant for the consolidation check dam Despite the ldquoDoesnot applyrdquo option VLG and TLG still preferred not to an-swer

33 Common aspects for the functional statusparameter C and synthesis

In questions C1 and C2 for bridges and check dams partici-pants only distinguished the upstream and downstream loca-tion from the field inspection (test 1) That is comparing tothe pretest results which was the preliminary inspection test



Figure 4 Relative frequencies for Vs and Ts of learning groups (LGs) and control groups (CGs) Parameter A and B in inspection tests forcheck dams

to the learning session For the pretests mode scores wereonly assessable during the field inspections (test 1) due tothe participantsrsquo difficulty in compiling the form in front ofa poster (pretest) Similar performance of the pretests holdsfor check dam 1 and bridge 1 indistinctly of the parameterinspected (Figs2ndash6)

When looking at the overall agreement the most subjec-tive aspects were the level of erosion at the stream bank (pa-rameter C) and synthesis of the inspection For the first onethe description coming along with the rating classes shouldbe improved The scheme that supports this aspect shouldalso be adjusted in order to minimize misunderstandings be-tween the left and right bank For the latter the synthesisshould remain optional for volunteer inspectors However Tsdid not agree for all structures Then participants still requirea short handout which can be taken with them to the field tosupport their choices

4 Discussion

First-level inspection of hydraulic structures is a citizen sci-ence project that aims to support decisions about obstruc-tions of hydraulic structures or to pre-screen problems for

more technical and detailed inspections Potential use espe-cially holds on dynamic environments with a large number ofhydraulic structures or where financial and human resourcesare limited (Danielsen et al 2005 Holub and Huumlbl 2008 deJong 2013)

In this research we combined rating scales with ordinalscores to get an indication on the functional status Howeverwe still acknowledge the usefulness of photo and videos tosupport detailed descriptions (Dirksen et al 2013 Yetman2002) Regardless of their importance not all volunteers tookphotographs They expressed difficulties to relate the photorecord in the form Then we made distinction among vol-unteer data with those obtained by professional staff Refer-ring to participantsrsquo characteristics both the Ts and Vs weremostly adult men However Vs were younger participantsas groups included students Ts generally had higher level ofeducation than Vs This can be explained again by the pres-ence of students but also by the fact that volunteering for civilprotection does not require high level of education Most Tsand half of the Vs were local inhabitants of the Fella BasinVs who do not live there were mostly the students who livelargely in the FVG region

Compared with data collected by Ts Vs often had highervariance Differences in accuracy can be explained from their



Figure 5Relative frequencies for Vs and Ts of learning groups (LGs) and control groups (CGs) Parameter C and Synthesis of the inspectiontests for bridges

hazard experience and preliminary knowledge for the inspec-tion of hydraulic structures The experience with natural haz-ards varied greatly for all the groups some participants hadnever experienced natural hazards when some had more than10 times Not surprisingly Ts had very good a prior knowl-edge for debris-flow phenomena functionality of check damsand culverts as well as on emergency security guidelinesVs scored lower especially in the VLG

Survey procedures were discussed with the LG Thusforms may be differently interpreted when comparing be-tween the TLG and TCG Similarly the preliminary knowl-edge of geosciences students may also influence their per-formance in the VCG That is by comparing with the VLGwhich were mostly composed of citizen volunteers of civilprotection Differences between Vs and Ts can evidencedlack of training and unfamiliarity with survey protocols

We found that the use of rating scales with a range in pre-cision of one level could cope with some variance Previousstudies indicated the advantages of rating classes (Yetman

2002) and the combination of classes depending on the ques-tions to be answered (Rinderer et al 2012) However visualinspections are subjected to various sources of biases bothfor the volunteers and technicians For example limitationson the accuracy for the recognition of defects precision todescribe defects according to a rating scale and completenessof the inspection reports (Gouveia et al 2004 Dirksen et al2013) Therefore it was useful to include unspecified optionsdistinguishing limitations such as water level and inspectionconditions which are also complementary information to an-alyze the reports

Furthermore we referred to potential improvements insurvey procedures Rating scales should consider all possiblefunctional conditions only when distinction among differentconcerns is possible Improvements in the forms are still re-quired For example a separated form could be defined forculverts to address specific aspects of such structures (Najafiand Bhattachar 2011) However surveys procedures shouldremain as simple as possible



Figure 6Relative frequencies for Vs and Ts of learning groups (LGs) and control groups (CGs) Parameter C and Synthesis of the inspectiontests for check dams

Despite the needs for improvement several commentsproved the utility of the activity ldquoas a good initiative to in-struct volunteers on the observation of the territory with pre-ventive scope it joined theory and practice together on thefield it helped to understand and inspect the functionality ofthe structurerdquo

Nevertheless iterative design and additional testing is re-quired before making a perennial activity from this pilotFirst improvements on the inspection procedures should beseparately tested with technicians to improve robustness ofmethods Thus we could validate the iterative design withinthe reference group Then we could discuss procedures withthe technicians for later examination of data and the use fordecision-making

The involvement of a mixed group of participants was in-teresting for knowledge exchange particularly in the outdoorsession However it also facilitated interaction during the in-spection tests which was ideally not desired Then replica-tion exercises should be carried out on a separate day foreach participantrsquos group to improve consistency on the meth-

ods Finally participants should have feedback on the qualityevaluation following every inspection campaign It may con-tribute to maintaining data quality during the design phasebut also on a long-term basis

Moreover citizen-based approaches require the effec-tive combination of two practical aspects in order to befully useful recruiting and training strategies Then quality-assurance methods for data collection comparison and ex-amination (Crall et al 2011 Riesch and Potter 2014)

From the social perspective a cornerstone beyond our re-search scope is the increasing volunteersrsquo awareness of thewater-sediment processes being addressed (Couvet et al2008) From the technical perspective geoinformatic toolsand mobile devices could facilitate data collection accessand validation (Newman et al 2012) Data management sys-tems could support technicians to compare and use collecteddata for later examination

In addition participants could exploit smartphone applica-tions for form compiling completeness checking data trans-ferring and photo record Additional tools could be included



Table 5Evaluation of scores for the data collected in the inspection tests for bridges The values in bold distinguish the lowest performance

Participantrsquos number 36 31 31Component questions per parameter Test 1 Bridge 1 Test 2 Bridge 2 Test 3 Bridge 3

(Rating classes assigned scores) X SD Us X SD Us X SD Us

(A) Condition of the structure

A1 Erosion at the pillar or abutment 13plusmn10 3 25 plusmn16 19 10 ndash 3 (3 classes 1 4 and 5)A2 Natural jumps created by the stream 12plusmn09 ndash 17 plusmn15 13 11 plusmn07 3 (2 classes 1 and 5)A3 Other damage at the foot of the structure 10 ndash 3 19plusmn17 13 10 ndash 10 (2 classes 1 and 5)

(B) Level of obstruction at the structure

B1 Obstruction upstream 13 plusmn05 ndash 17 plusmn09 3 18 plusmn09 ndash(5 classes 1 2 3 4 and 5)B2 Obstruction downstream 13 plusmn05 ndash 17 plusmn08 3 15 plusmn06 3 (5 classes 1 2 3 4 and 5)B3 Islands with vegetation (shrub) or other 17plusmn15 ndash 29 plusmn20 ndash 18 plusmn16 ndashman-made obstructions (2 classes 1 and 5)

(C1) Presence of protection works within 20 m upstream and downstream

C11 Left bank ndash upstream 49 plusmn07 3 16 plusmn13 13 44 plusmn12 ndash(3 classes 1 4 and 5)C12 Left bank ndash downstream 48 plusmn07 ndash 16 plusmn14 3 44 plusmn10 3 (3 classes 1 4 and 5)C13 Right bank ndash upstream 50 plusmn02 6 35 plusmn19 3 31 plusmn18 3 (3 classes 1 4 and 5)C14 Right bank ndash downstream 48plusmn07 ndash 42 plusmn14 10 45 plusmn08 ndash(3 classes 1 4 and 5)

(C2) Level of erosion at the stream bank within the same distance

C21 Left bank ndash upstream 11 plusmn03 19 16 plusmn06 19 16 plusmn09 3 (5 classes 1 2 3 4 and 5)C22 Left bank ndash downstream 11 plusmn03 14 24 plusmn13 16 17 plusmn10 3 (5 classes 1 2 3 4 and 5)C23 Right bank ndash upstream 11 plusmn02 17 13 plusmn05 10 20 plusmn09 3 (5 classes 1 2 3 4 and 5)C24 Right bank ndash downstream 11plusmn03 22 20 plusmn08 6 23 plusmn13 ndash(5 classes 1 2 3 4 and 5)

Synthesis of the inspection

(4 classes 1 2 4 and 5) 15plusmn11 19 26 plusmn15 6 32 plusmn18 6

X average ordinal score SD standard deviation Us relative frequency of unspecified answers

such as an embedded glossary or a systematic tag GPS signalcoverage of mobile devices is especially limited in mountaincatchments Therefore a known ID of the structure may stillbe relevant Future research using such applications shouldaddress data-quality requirements Usability should exploreadvantages for the diversity of volunteers getting involved(Newman et al 2010)

5 Conclusions

Results showed that citizen volunteers could carry out first-level inspections with comparable performance to techni-cians Differences among the 11 technicians and 25 volun-teers do not have a high statistical significance when distinc-tion is done among control and learning groups Howeverkey points can still be extracted from this data set Those con-siderations are relevant for the use of volunteersrsquo data on the



Table 6Evaluation of scores for the data collected in the inspection tests for check dams The values in bold distinguish the lowest perfor-mance

Participantrsquos number 35 31 31Component questions per parameter Test 1 Check dam 1 Test 2 Check dam 2 Test 3 Check dam 3



A1 Stream flow passing where it should be 11plusmn07 ndash 20 plusmn15 ndash 20 plusmn14 ndash(3 classes 1 4 and 5)A2 Status of the check dam 22plusmn15 9 10 plusmn02 6 11 plusmn03 ndash(5 classes 1 2 3 4 and 5)A3 Visibility of the basis of the structure 22 plusmn17 23 13 plusmn06 3 13 plusmn06 3 (5 classes 1 2 3 4 and 5)A4 Protection for dowmstream scouring 47 plusmn09 11 32 plusmn06 32 19 plusmn10 16 (3 classes 3 1 and 5)


B1 At the opening of the check dam if any 10 ndash 26 44 plusmn08 ndash 21 plusmn05 3 (5 classes 1 2 3 4 and 5)B2 Upstream in the retention basin if any 12plusmn08 26 40 plusmn14 13 12 plusmn05 16 (3 classes 1 3 and 5)B3 Downstream obstruction 11 plusmn02 9 25 plusmn13 3 18 plusmn09 ndash(5 classes 1 2 3 4 and 5)


C11 Left bank ndash upstream 17 plusmn15 6 13 plusmn09 13 44 plusmn08 6 (3 classes 1 4 and 5)C12 Left bank ndash downstream 49 plusmn03 3 45 plusmn08 3 46 plusmn10 ndash(3 classes 1 4 and 5)C13 Right bank ndash upstream 46plusmn12 6 11 plusmn06 16 40 plusmn13 ndash(3 classes 1 4 and 5)C14 Right bank ndash downstream 49plusmn03 3 29 plusmn17 3 46 plusmn10 3 (3 classes 1 4 and 5)


C21 Left bank ndash upstream 14 plusmn07 26 18 plusmn12 39 13 plusmn04 3 (5 classes 1 2 3 4 and 5)C22 Left bank ndash downstream 12 plusmn05 11 15 plusmn09 19 13 plusmn04 ndash(5 classes 1 2 3 4 and 5)C23 Right bank ndash upstream 11 plusmn03 14 19 plusmn12 39 16 plusmn08 6 (5 classes 1 2 3 4 and 5)C24 Right bank ndash downstream 11plusmn03 17 26 plusmn14 23 13 plusmn05 ndash(5 classes 1 2 3 4 and 5)




functional status of hydraulic structures It may also providesome guidance to researchers and practitioners interested incitizen-based data

1 Volunteers could carry out first-level inspections withcomparable performance to technicians but with a pre-required range in precision However survey proce-

dures that are clear enough require iterative design andtesting to avoid uncertainty and misunderstandings

2 In spite of the need to standardize reports unstructureddata such as comments and pictures are still requiredby managers to validate completeness and precision ofvolunteersrsquo data Then the systematic tagging and ref-erencing of that data is crucial



Table 7Feedback of participants to initiate corrective actions in survey procedures

Feedback form Comments received Corrective actions

Was the inspection form clear Protection works for scouring should Rating classes and schemes will be adaptedenough to carry out the inspection be inspected not only for check dams according to participantsrsquo comments and

but also for bridges when it applies recommendations from the results sectionFor bridges obstructions in the floodplain Brief guidelines and glossary must be providedshould be also reported together with the inspection formUpstream obstruction should be reported for openand consolidation check dams

Did you find the options When possible rating scales with three or two classes To avoid misunderstandings the question regardingprovided in the form useful should be extended to rate all possible status the presence of protection works will better refer toto answer the questions Presence of human infrastructure should be their length within the control distance

open question to report other infrastructure The form will emphasize to report the infrastructurebesides roads and buildings that may be affected in case of high water levels

Which aspects you did not All structures to inspect should have available Information regarding the type of structurelike about the activity information for the function type will be always precompiled in the form

Participants with technical background The learning should start directly withconsidered the indoor session long while the outdoor session and finishes withcitizen-volunteers requested more time the indoor sessionto better understand the theory and The indoor session will be carried outto carry out the inspections separately for each group of participantsThe inspection in front of the poster Interaction between groups will be limitedcould be better used after both theory to the outdoor sessionand practice have been explained

3 Unspecified answers may persist according to the com-plexity of connected elements to the structure and theunexpected conditions for the inspection Rating classesshould specify when water or sediment did not allow theassessment However other options should be limited tofacilitate the later examination of data

4 Volunteer ratings should be considered as first-level as-sessment Managers could combine these ratings to getindexes on the status of the structure at parameter levelHowever an indication of the overall completeness perparameter would still be needed for the later examina-tion

5 The use of scores to convert volunteer ratings is im-portant for getting an indication of the functional sta-tus Since the rating scales are expressed in linguisticterms ratings could be converted into numbers by usinga fuzzy set theory instead of ordinal scores Conversionof the volunteersrsquo data using scales of fuzzy terms couldhandle the pre-required ranges in precision (eg fromlow to very low)

Important considerations to improve and promote citizenscience projects are firstly related to limitations on citizeninvolvement due to different culture of volunteer activitiesand interest in participating Secondly training is relevant forthe performance of volunteers but also for increasing aware-ness and preparedness on the causes and consequences of hy-dro meteorological hazards (Enders 2001) For the first onestudents are an alternative approach for the citizensrsquo recruit-ment where there is limited volunteer culture Universities

could involve students of geosciences or social sciences togain practical knowledge or better understand their territory(Savan et al 2003)

For the latter future research should test the effectivenessof the learning session according to differences in prelim-inary knowledge of participants In the study area volun-teer groups offered opportunities to carry out first-level in-spections However replication exercises are still needed inother study areas to improve the consistency and robustnessof the data evaluation In addition survey procedures couldbe adapted to other target groups eg last-year high schoolstudents who are not awareinvolved in management activi-ties That could be an alternative approach to enhance theirawareness of hydro-meteorological hazards



Appendix A

Table A1 Participantsrsquo characteristics for Vs and Ts of learninggroups (LGs) and control groups (CGs)

Participantrsquos TLG TCG VLG VCGnumber 4lowast 5 18 7

Age (years)

16ndash24 ndash ndash 7 425ndash44 2 2 7 245ndash64 2 3 4 1

Gender

Female ndash 1 3 2Male 4 4 15 5

Highest level of education

Elementary school ndash ndash 2 ndashCollege ndash ndash 4 2High school 1 1 10 3Technical degree ndash ndash 2 2Masterrsquos degree 2 3 ndash ndashPhD 1 1 ndash ndash

Period of residence in the Fella Basin

More than 20 years 3 4 7 5Never 1 1 11 2

Period of residence in the FVG region

Less than a year ndash ndash 2 ndash1ndash5 years ndash ndash 1 1More than 20 years 4 5 14 4Never ndash ndash 1 2

Table A1 Continued


Experiences with floods

Never 1 1 6 3Once ndash ndash 1 12ndash5 times 2 2 8 36ndash10 times ndash ndash 1 ndashMore than 10 times 1 2 1 ndashNo data ndash ndash 1 ndash

Experiences with debris flows

Never 1 1 8 4Once ndash ndash ndash ndash2ndash5 times 2 2 4 36ndash10 times 1 1 4 ndashMore than 10 times ndash 1 1 ndashNo data ndash ndash 1 ndash

Experiences with landslides

Never ndash 2 7 3Once ndash ndash 2 22ndash5 times 2 1 3 26ndash10 times ndash 1 3 ndashMore than 10 times 2 1 2 ndashNo Data ndash ndash 1 ndash

A prior knowledge ( of correct answers on 20 questions)

0ndash25 ndash ndash 1 ndash26ndash50 ndash ndash 3 ndash51ndash75 ndash ndash 8 676ndash100 4 5 6 1

lowast 2 TLG did not fill the questionnaire Table summarize data for 34participants



The Supplement related to this article is available onlineat doi105194nhess-14-2681-2014-supplement

AcknowledgementsThis research was conducted in collaborationwith civil protection technical services and the local authoritiesof FVG Students of the Universitagrave degli Studi di Trieste werealso participants of the research Authors would like to thankRoxana Ciurean and Chiara Calligaris for their valuable supportduring the fieldwork period in terms of organizational aspectsWe would also like to thank reviewers for their comments thathelp to improve the manuscript This work is part of the MarieCurie ITN changing hydro-meteorological risks as analyzed bya new generation of European scientists (CHANGES project)which is funded by the European communityrsquos seventh frameworkprogramme FP72007-2013 under grant agreement no 263953

Edited by F GuzzettiReviewed by two anonymous referees

References

ADBVE (Autoritagrave di bacino dei fiumi dellrsquoAlto Adriatico) Pro-getto di Piano Stralcio per lrsquoAssetto Idrogeologico del bacinoidrogeografico del fiume Fella PAIndashFELLA available athttppaiadbveitPAI_Fellaindex_fellahtml last access 7 July 20142012 (in Italian)

Bjorkland R Pringle C and Newton B A Stream Visual Assess-ment Protocol (SVAP) for Riparian landowners Environ MonitAssess 68 99ndash125 doi101023A1010743124570 2001

Bonney R Cooper C B Dickinson J L Kelling S PhillipsT Rosenberg K V and Shirk J Citizen science a develop-ing tool for expanding science knowledge and scientific literacyBioScience 59 977ndash984 doi101525bio200959119 2009

Bordogna G Carrara P Criscuolo L Pepe M and RampiniA A linguistic decision making approach to assess the qualityof volunteer geographic information for citizen science Infor-mation Sciences 258 312ndash327 doi101016jins2013070132014

Borga M Boscolo P Zanon F and Sangati M Hydrom-eteorological Analysis of the 29 August 2003 Flash Floodin the Eastern Italian Alps J Hydrometeorol 8 1049ndash1067doi101175JHM5931 2007

Brandon A Spyreas G Molano-Flores B Caroll C and El-lis J Can volunteers provide reliable data for forest vegetationsurveys National Areas Journal 23 254ndash262 2003

Burke Engineering C B Indiana Drainage Handbook an Ad-ministrative and Technical Guide for Activities within Indi-ana Streams CBBEL Indianapolis Indiana available athttpwwwingovdnrwaterfilesallhbookpdf last access 4 July2013 1999

Calligaris C and Zini L Debris Flow Phenomena A ShortOverview in Earth Sciences edited by Imran Ahmad Dar In-Tech ISBN 978-953-307-861-8 doi10577229786 2012

Cifelli R Doesken N Kennedy P Carey L D RutledgeS A Gimmestad C and Depue T The Community Collab-orative Rain Hail and Snow Network informal education forscientists and citizens B Am Meteorol Soc 86 1069ndash1077doi101175BAMS-86-8-1069 2005

Conrad C and Hilchey K A review of citizen scienceand community-based environmental monitoring issuesand opportunities Environ Monit Assess 176 273ndash291doi101007s10661-010-1582-5 2011

Couvet D Jiguet F Julliard R Levrel H and Teysse-dre A Enhancing citizen contributions to biodiversity sci-ence and public policy Interdiscipl Sci Rev 33 95ndash103doi101179030801808X260031 2008

Crall A W Newman G J Jarnevich C S Stohlgren T JWaller D M and Graham J Improving and integrating data oninvasive species collected by citizen scientists Biol Invasions12 3419ndash3428 doi101007s10530-010-9740-9 2010

Danielsen F Burgess N D and Balmford A Monitoring mat-ters examining the potential of locally-based approaches Bio-divers Conserv 14 2507ndash2542 doi101007s10531-005-8375-0 2005

de Jong C Linking ICT and society in early warning and adap-tation to hydrological extremes in mountains Nat HazardsEarth Syst Sci 13 2253ndash2270 doi105194nhess-13-2253-2013 2013

Devictor V Whittaker R J and Beltrame C Beyond scarcitycitizen science programmes as useful tools for conservation bio-geography Divers Distrib 16 354ndash362 doi101111j1472-4642200900615x 2010

Dirksen J Clemens F H L R Korving H Cherqui F Le Gauf-fre P Ertl T Plihal H Muumlller K and Snaterse C T M Theconsistency of visual sewer inspection data Struct InfrastructE 9 214ndash228 doi101080157324792010541265 2013

Enders J Measuring community awareness and preparedness foremergencies Australian Journal of Emergency Management 1652ndash58 2001

Engel S R and Voshell Jr J R Volunteer biological monitor-ing can it accurately assess the ecological condition of streamsAmerican Entomologist 48 164ndash177 2002

EPA Volunteer Stream Monitoring A Methods Manual UnitedStates Environmental Protection Agency Washington DC avail-able at httpwaterepagovtyperslmonitoringupload2002_08_13_volunteer_stream_streampdf last access 9 February2014 1997

European Commission Directive 200760EC of the EuropeanParliament and of the Council of 23 October 2007 on theAssessment and Management of Flood Risks available athttpeur-lexeuropaeulegal-contentENTXTuri=CELEX32007L0060 last access 9 February 2014 2007

European Commission Civil Protection ndash Community Co-operation in the Field of Civil Protection Humanitarian Aidand Civil Protection available athttpeceuropaeuechofilescivil_protectionvademecummenu3html last access 9 Febru-ary 2014 2012

Flanagin A J and Metzger M J The credibility of vol-unteered geographic information GeoJournal 72 137ndash148doi101007s10708-008-9188-y 2008



Fore L S Paulsen K and OrsquoLaughlin K Assessing the perfor-mance of volunteers in monitoring streams Freshwater Biol 46109ndash123 doi101111j1365-2427200100640x 2001

Galloway A W E Tudor M T and Haegen W M V The relia-bility of citizen science a case study of Oregon White Oak standsurveys Wildlife Soc B 34 1425ndash1429 doi1021930091-7648(2006)34[1425TROCSA]20CO2 2006

Gollan J Bruyn L L Reid N and Wilkie L Can volun-teers collect data that are comparable to professional scien-tists a study of variables used in monitoring the outcomesof ecosystem rehabilitation Environ Manage 50 969ndash978doi101007s00267-012-9924-4 2012

Gommerman L and Monroe M C Lessons Learned from Evalu-ations of Citizen Science Programs School of Forest Resourcesand Conservation Florida Cooperative Extension Service In-stitute of Food and Agricultural Sciences (IFAS) Universityof Florida (FOR291) available athttpedisifasufledufr359last access 9 February 2014 2012

Goodchild M F Citizens as sensors the world of volunteeredgeography GeoJournal 69 211ndash221 doi101007s10708-007-9111-y 2007

Goodchild M F and Li L Assuring the quality of volun-teered geographic information Spatial Statistics 1 110ndash120doi101016jspasta201203002 2012

Gouveia C and Fonseca A New approaches to environmentalmonitoring the use of ICT to explore volunteered geographicinformation GeoJournal 72 185ndash197 doi101007s10708-008-9183-3 2008

Gouveia C Fonseca A Cacircmara A and Ferreira F Promotingthe use of environmental data collected by concerned citizensthrough information and communication technologies J Envi-ron Manage 71 135ndash154 doi101016jjenvman2004010092004

Holub M and Fuchs S Mitigating mountain hazards in Aus-tria ndash legislation risk transfer and awareness building NatHazards Earth Syst Sci 9 523ndash537 doi105194nhess-9-523-2009 2009

Holub M and Huumlbl J Local protection against mountain hazardsndash state of the art and future needs Nat Hazards Earth Syst Sci8 81ndash99 doi105194nhess-8-81-2008 2008

Hudson-Smith A Batty M Crooks A and Milton R Map-ping for the masses accessing Web 20 through crowdsourc-ing in Working Paper Series ndash University College of Londonfor Advanced Spatial Analysis Centre for Advanced SpatialAnalysis University College London London UK available athttpeprintsuclacuk15198115198pdf last access 9 Febru-ary 2014 Paper 143 1ndash18 2008

Jakob M and Hungr O Debris Flow Phenomena Praxis Berlin2005

Jordan R C Brooks W R Howe D V and Ehrenfeld J GEvaluating the performance of volunteers in mapping invasiveplants in public conservation lands Environ Manage 49 425ndash434 doi101007s00267-011-9789-y 2011

Mazzorana B Huumlbl J Zischg A and Largiader A Modellingwoody material transport and deposition in alpine rivers NatHazards 56 425ndash449 doi101007s11069-009-9492-y 2010

Molinari D Menoni S Aronica G T Ballio F Berni N Pan-dolfo C Stelluti M and Minucci G Ex post damage assess-

ment an Italian experience Nat Hazards Earth Syst Sci 14901ndash916 doi105194nhess-14-901-2014 2014

Najafi M and Bhattachar D V Development of a culvert inven-tory and inspection framework for asset management of roadstructures Journal of King Saud University - Science 23 243ndash254 doi101016jjksus201011001 2011

Newman G Zimmerman D Crall A Laituri M Graham Jand Stapel L User-friendly web mapping lessons from a cit-izen science website Int J Geogr Inf Sci 24 1851ndash1869doi101080136588162010490532 2010

Newman G Wiggins A Crall A Graham E Newman S andCrowston K The future of citizen science emerging technolo-gies and shifting paradigms Front Ecol Environ 10 298ndash304doi101890110294 2012

Nicholson E Ryan J and Hodgkins D Community data-wheredoes the value lie Assessing confidence limits of communitycollected water quality data Water Sci Technol 45 193ndash2002002

Ohio Department of Transportation Manual of Bridge InspectionOhio Department of Transportation Federal Highway Adminis-tration Ohio 2010

Province of British Columbia Flood Protection Works Inspec-tion Guide Water Management Branch Public Safety SectionBritish Columbia Canada 2000

Provinzia Autonoma di Bolzano ndash Alto Adige Bestandsaufnahmevon Wasserbauwerken der Wildbachverbauung EF 30 ndash EF30Sperrenevaluierung Quick Version 2006 (in German)

Protezione Civile della Regione FVG Formazione Campus Vir-tuale available athttpwwwprotezionecivilefvgitProtCivdefaultaspx81-formazionehtm last access 7 July 2014 2009

Remaicirctre A Malet J-P and Maquaire O Morphology and sed-imentology of a complex debris flow in a clay-shale basin EarthSurf Proc Land 30 339ndash348 doi101002esp1161 2005

Riesch H and Potter C Citizen science as seen by scientistsmethodological epistemological and ethical dimensions Pub-lic Underst Sci 23 107ndash120 doi10117709636625134973242014

Rinderer M Kollegger A Fischer B M C Staumlhli M and Seib-ert J Sensing with boots and trousers ndash qualitative field obser-vations of shallow soil moisture patterns Hydrol Process 264112ndash4120 doi101002hyp9531 2012

Savan B Morgan A J and Gore C Volunteer environmen-tal monitoring and the role of the universities the case ofcitizensrsquo environment watch Environ Manage 31 561ndash568doi101007s00267-002-2897-y 2003

Seeger C J The role of facilitated volunteered geographicinformation in the landscape planning and site design pro-cess GeoJournal 72 199ndash213 doi101007s10708-008-9184-22008

Servizio Forestale FVG Catasto Opere Iidraulico Forestali Man-uale Tecnico Allegato A Schede di Rilevo 2002

Snaumlll T Kindvall O Nilsson J and Paumlrt T Evaluating citizen-based presence data for bird monitoring Biol Conserv 144804ndash810 doi101016jbiocon201011010 2011

Tweddle J Robinson L Pocock M and Roy H Guide to Cit-izen Science Developing Implementing and Evaluating CitizenScience to Study Biodiversity and the Environment in the UKNatural History Museum London 2012



United Nations Hyogo Framework for Action 2005ndash2015 Building the Resilience of Nations and Commu-nities to Disasters Kobe Japan 18ndash22 available athttpwwwunisdrorg2005wcdrintergoverofficial-docL-docsHyogo-framework-for-action-englishpdf last access9 February 2014 2005

Uzielli M Nadim F Lacasse S and Kaynia A MA conceptual framework for quantitative estimation of phys-ical vulnerability to landslides Eng Geol 102 251ndash256doi101016jenggeo200803011 2008

von Maravic P Evaluation of the Physical Vulnerability of CheckDams Exposed to the Impact of Torrential Processes ThroughExperimental Analysis Msc thesis University of Padua Facultyof Agriculture Departments of Land and Agro-forestry Systems2010

Yetman K T Using Marylandrsquos stream corrido survey to prioritizewatershed restoration efforts J Am Water Resour As 38 905ndash914 doi101111j1752-16882002tb05533x 2002



sediment processes also affect the functional status of hy-draulic structures and vice versa (Holub and Huumlbl 2008)

The impact (ie damage) is evident to structures fordebris-flow control such as check dams Evidence is alsofound in the potential aggravation of flood hazard at the lo-cation of bridges and culverts due to blocking material suchas debris large wood and other residues (Mazzorana et al2010) Stability of protection works is often threatened bythe erosion level at stream banks

The need to enhance data-collection approaches to supportrisk-management strategies is widely acknowledged (egMolinari et al 2014) Besides situations where financial andhuman resources are limited scientific monitoring may besubject to additional complexity under dynamic environmen-tal conditions or remote settings (de Jong 2013)

Moreover frequent inspection of hydraulic structures isespecially important in mountain basins Therefore oppor-tunities in promoting citizen science projects stem from theincreasing frequency timeliness and coverage of surveillanceactivities (Flanaging and Metzger 2008) To be useful sur-vey procedures should be tested and adapted according toquality requirements of decision-makers (Bordogna et al2014 EPA 1997 Goodchild and Li 2012 Gouveia and Fon-seca 2008)

Experiences of citizen science include data collection re-garding water quality and biological aspects (Engel andVoshell Jr 2002 Fore et al 2001 Nicholson et al 2002)forestry and ecosystem rehabilitation (Brandon et al 2003Gollan et al 2012) biodiversity (Snaumlll 2011) stream mon-itoring (Bjorkland et al 2001 Yetman 2002) and hy-drological processes (Cifelli et al 2005 Rinderer et al2012) Despite the variety of projects limited researchhas been devoted to evaluating the quality of citizen-baseddata (Danielsen et al 2005) Furthermore scientists anddecision-makers have a general lack of confidence due to thelimited accuracy non-comparability and incompleteness ofcitizen-collected data (Riesch and Potter 2014 Conrad andHilchey 2011)

In spite of the challenges for citizen involvement preci-sion and completeness of largely collected data depends onthe exhaustiveness of the inspection procedures (Gallowayet al 2006) Therefore training activities are often requiredbefore starting the inspection campaigns However the ex-tension of these training sessions should consider availabletime number and type of participants (Tweddle et al 2012)To that endJordan et al(2011) suggested that identifica-tion of technical data should be restricted while more gen-eral indicators can still be accurately obtained Those indica-tors may be quantitative and qualitative aspects that are eas-ily recognizable from visual inspections (Gommerman andMonroe 2012 Gouveia et al 2004) Then qualitative fieldmethods are generally based on rating scales to report in-spected conditions

This study considers regular inspections with citizen-volunteer groups that are promoted by civil protection andlocal authorities of Friuli Venezia Giulia (FVG) Italy We in-volved 11 technicians and 25 volunteers on a data-collectionexercise Participants were invited according to their lo-cation Thus 15 out of 25 volunteers were members ofcivil-protection groups in neighboring municipalities In all10 university students were also volunteers within a supple-mentary academic activity

In this paper we evaluate data quality on preliminary in-spections of bridges and check dams Therefore we addressthe following research questions (1) how well were partici-pants able to report on the functional status by distinguishingbetween available rating classes (2) How effectively weredata collected by volunteers compared to those collected bytechnicians (3) How can survey procedures be improvedTo that end Sect2 describes the methodology for data col-lection In Sect3 we evaluate data quality by their accuracyprecision and completeness Finally we highlight in the dis-cussionrsquos and conclusionrsquos (Sects4 and5) key points for thepractical use of citizen-based data

2 Methods

In the first step of the methodology we defined target groupsThe participantsrsquo groups comprised of volunteers and tech-nicians to evaluate the data quality In the second step twoinspection forms were designed for bridges and check damsThe forms were created to carry out inspections with trainedvolunteers

Despite available procedures for technicians the volun-teersrsquo involvement demands more structured and simplerforms to inspect the functional status Similar toYetman(2002) we used rating scales to standardize collected data indistinguishing minor problems from more serious concernsThe rating scales included visual schemes to guide inspec-tors In addition the latter were required to take a photo tosupport their choices

Finally we organized a data-collection exercise to carryout first-level inspection of six structures hereafter referredto as ldquoinspection testsrdquo There was 1 day for the training ses-sion and 1 day for the inspection tests Participants were di-vided between control and learning groups to identify poten-tial improvements in survey procedures

21 Participantsrsquo groups

Citizens were involved in the form of civil-protection volun-teers due to safety limitations and accessibility to hydraulicstructures in mountain catchments Citizens enrolled as civil-protection volunteers traditionally received formative infor-mative and safety procedures while specialized training is se-lectively provided (Protezione Civile FVG 2009)


















































































(Check dam 2 and 3)

































4 Discussion








































5 Conclusions








































Appendix A



Age (years)


Gender








Table A1 Continued
















References
















































































































































(Check dam 2 and 3)

































4 Discussion








































5 Conclusions








































Appendix A



Age (years)


Gender








Table A1 Continued
















References




























































































4 Discussion








































5 Conclusions








































Appendix A



Age (years)


Gender








Table A1 Continued
















References


































































































5 Conclusions








































Appendix A



Age (years)


Gender








Table A1 Continued
















References





































































































Appendix A



Age (years)


Gender








Table A1 Continued
















References




































































References
































































evaluating data quality collected by volunteers for ﬁrst ... · teers due to safety limitations...

Documents