technical guidance for on-the-spot checks (otsc) and … · this publication is a technical report...
TRANSCRIPT
Philippe LOUDJANI Par ASTRAND Vincenzo ANGILERI Dominique FASFENDER Pavel MILENOV Cozmin LUCAU
Common Technical Specifications
For the 2016 campaign of On-
The-Spot Checks
TECHNICAL GUIDANCE
FOR ON-THE-SPOT CHECKS (OTSC) AND AREA MEASUREMENT ACCORDING TO ART. 24, 25, 26, 27, 30, 31, 34, 35, 36, 37, 38, 39,
40, 41 OF REGULATION (EU) NO 809/2014 AS AMENDED BY REGULATION (EU) 2015/2333
second main title line third main title line <Main title, Verdana 28, line spacing 32 pt>
2016
DS-CDP-2016-03
This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science and
knowledge service. It aims to provide evidence-based scientific support to the European policy-making process.
The scientific output expressed does not imply a policy position of the European Commission. Neither the
European Commission nor any person acting on behalf of the Commission is responsible for the use which might
be made of this publication.
Contact information
Name: Philippe LOUDJANI
Address: Via Fermi 2749, TP266, 26b/038 I-21027 ISPRA (VA), ITALY
E-mail:[email protected]
Tel.: +39.0332.78.6160
JRC Science Hub
https://ec.europa.eu/jrc
JRCxxxxx
EUR xxxxx xx
PDF ISBN xxx-xx-xx-xxxxx-x ISSN xxxx-xxxx doi:xx.xxxx/xxxxxx
Print ISBN xxx-xx-xx-xxxxx-x ISSN xxxx-xxxx doi:xx.xxxxx/xxxxxx
Ispra, Italy: European Commission, 2016
© European Union, 2016
Reproduction is authorised provided the source is acknowledged.
How to cite: Author(s); title; EUR; doi
All images © European Union 2016,
Table of contents
1 CONTENTS
2 Overview ........................................................................................................ 5
3 Documentation, justification and decisions sharing .............................................. 8
4 Preliminary work ............................................................................................. 9
3.1 Define strategy ............................................................................................. 9
3.2 Sample selection ......................................................................................... 10
A. Random selection................................................................................. 10
B. Risk analysis and annual assessment ...................................................... 10
4.3 Selection of Control zones ......................................................................... 13
4.4 Suggestion of an iterative procedure for selection of control zones and samples
14
5 Preparation of checks ..................................................................................... 15
5.1 Reception of applications and data entry ..................................................... 15
5.2 Collection of LPIS data and other ancillary documents .................................. 15
5.3 Provision of satellite images ...................................................................... 16
5.4 Management of satellite image acquisition ................................................... 16
A) Use of Copernicus datasets ......................................................................... 17
5.5 Acquisition of aerial imagery (if applicable) .................................................. 18
5.6 Processing of satellite images and aerial photographs ................................... 19
A. Geometric correction ............................................................................ 19
B. Radiometric correction .......................................................................... 21
5.7 Ground data collection .............................................................................. 21
5.8 Production of controller guide (CAPI guide and field control) .......................... 22
5.9 GNSS and ortho image validation ............................................................... 22
6 Perform checks.............................................................................................. 23
6.1 Check at parcel level ................................................................................... 23
A. Parcel location ..................................................................................... 23
B. Area check .......................................................................................... 23
C. Material to be used for parcel area measurements ................................... 25
D. When to perform parcel area measurement? ........................................... 28
E. Determination of the parcel area, use of the technical tolerance ................ 28
F. Minimum size ...................................................................................... 28
G. Parcel area measurement in the frame of “greening” ................................ 28
H. Afforested areas - area measurements ................................................... 29
I. Diagnosis and codes for area measurement check ........................................ 29
J. Capping to the reference area .................................................................... 30
6.2 Land use / Land cover check ........................................................................ 30
A. Use of ortho-rectified imagery (CwRS) .................................................... 31
B. Codes for diagnosis .............................................................................. 33
C. Rapid field visits .................................................................................. 36
D. Categorisation at scheme group level ..................................................... 36
E. Categorisation at dossier level ............................................................... 37
F. Final diagnosis at the dossier level ......................................................... 38
6.3 Production of OTSC report ............................................................................ 40
7 Analysis of the campaign – Act for the next campaign ........................................ 40
7.1 Quality Controls .......................................................................................... 40
7.2 Study of residual errors ........................................................................ 42
7.3 Analysis of campaign ................................................................................... 42
8 Annex 1: Details of image radiometric correction ............................................... 44
9 Annex 2: JRC "Area measurement tool validation method" .................................. 50
10 Annex 3: Recommended means of analysis of residual errors ........................... 66
A. Testing the significance of a random error rate on one control zone ........... 66
B. Comparing random error rates from different control zones ....................... 67
C. Comparing two error rates: paired case .................................................. 67
1
Pre-amble
This document, which has been prepared by the European Commission (Joint
Research Centre, JRC) in close collaboration with DG AGRI, describes the
Commission technical guidelines for the 2016 campaign of the On-The-Spot
Checks (OTSC). Member States may use these guidelines as Common Technical
Specifications of the Invitation To Tender (ITT) published by Member States
requiring an external remote sensing contractor. It complements the guidance
document for OTSC and area measurement: DSCG/2016/32 Final- Rev 2 final.
The document aims to describe the technical tasks that the Administrations of the
Member States are responsible for and for which some parts may be entrusted to
contractors. For the sake of completeness, however, the technical context of the
work requires some descriptions of the role and responsibilities of both the
Administration and the Commission. Some of the technical details may seem
exhaustive, but are included to allow Member State Administrations, and possible
contractors, the best possible chance to estimate the expected workloads.
Furthermore, as a common document, it has to be inclusive of all the possible
choices, options and alternatives that are used in the Member States for their On-
The-Spot Checks.
This document provides technical guidance on how to perform On-The-Spot
Checks in general. However, since the majority of the European Union Member
States is using Remote Sensing to control at least a part of the subsidies for the
agricultural areas funded by the EAGF and EAFRD, most of the information
provided concern the implementation of the so-called “Control with Remote
Sensing” (CwRS).
This document has to be complemented by a separate compulsory “National
Addendum”, which describes the choices and alternatives applicable in the
respective Member State. The information given in this “National Addendum” must
be taken into account in the bidders’ reply to ITT. This is all the more necessary
as different schemes coexist in the EU 28 which will be applied with different
models and variants. Since these Common Technical Specifications do not take
into account all particular situations in the different Member States, derogations
from particular rules indicated in this document should be introduced in the
National Addendum.
The information in this document is up-to-date with the existing EU regulations
that are applicable at the time of writing. It is the MS Administrations and the
contractors’ responsibility to be aware of other general or specific regulations
applicable in their respective Member States.
Technical Recommendations regarding the different phases of the work are also
available on the WikiCAP website1
1 https://marswiki.jrc.ec.europa.eu/wikicap/index.php/Main_Page
2
The role of the Commission in a possible procurement procedure for a contractor
to perform OTSC work, is strictly restricted to the technical support required to
compile the ITT document. The selection, award and follow-up of any contract
following from this open procedure is the sole responsibility of the awarding
authority in the respective Member States.
While the Commission has attempted to make the information contained in these
common technical specifications as accurate as possible, it does not warrant the
accuracy of the information contained or embodied in the document. The
Commission does not warrant or make any representations as to the accuracy of
the information contained in the National Addenda produced by respective
Member States. Contracts awarded are the sole responsibility of the awarding
Administrations in the respective Member States.
3
LIST OF ACRONYMS USED AND TERMINOLOGY FOR THE PURPOSE OF
THIS DOCUMENT
ACRONYMS
AECM = Agri-Environment-Climate Measures;
AECC = Agri-Environment-Climate Commitments;
ANC = Areas of Natural Constraints
AOI = Area Of Interest
BPS/SAPS/SFS = Basic Payment Scheme/ Single Area Payment Scheme as
referred to in Title III of Regulation (EU) No 1307/2013 and Small Farmers
Scheme as referred to in Title IV of the same Regulation;
BUNDLE = panchromatic and multispectral image bands (of an image
dataset)
CAPI = Computer Assisted Photo Interpretation;
CART = Classification And Regression Tree;
CD = Crop diversification;
CwRS = Control with Remote Sensing;
DEM= Digital Elevation Model;
DOP = Dilution Of Precision
DSM = Digital Surface Model;
EFA = Ecological focus areas as referred to in Article 46 of Regulation (EU) No
1307/2013 and its Delegated Regulation (EU) No 639/2014;
GAEC = good agricultural and environmental condition;
G4CAP = G4CAP is the Web-based application used to manage the whole
campaign workflow; https://g4cap.jrc.ec.europa.eu/G4CAP/
GNSS = Global Navigation Satellite System;
GSD = Ground Sampling Distance;
HDOP = Horizontal Dilution Of Precision
HR = High Resolution;
HHR = High High Resolution (increased quality compared to HR)
LPIS = identification system for agricultural parcels as referred to in Article 70 of
Regulation (EU) No 1306/2013;
MEA = Maximum Eligibility Area;
MS = Member States;
4
MSP = Multispectral (image)
OTSC = on-the-spot checks;
PA = Paying Agency
PAN = Panchromatic (image)
PDOP = Position Dilution of Precision
PG = Permanent Grassland as referred to in Art. 4(1)(h) of Regulation (EU) No
1307/2013;
PG-ELP = Permanent Grassland under Established Local Practices as referred to
in Art.4(1)(h) of Regulation (EU) No 1307/2013;
RA = Risk Analysis;
RAnF = Ratio of “Area not Found”;
RF = Risk Factors;
RFV = Rapid Field Visits;
RMSE = Root Mean Square Error;
RP = Reference Parcel;
RS = Remote Sensing;
SFS = Small farmers scheme as referred to in Title V of Regulation (EU) No
1307/2013;
SMR =Statutory management requirements;
SPS = Single Payment Scheme;
VCS = Voluntary coupled Support as referred to in Chapter 1 of title IV of
Regulation (EU) No 1307/2013;
VHR = Very High Resolution.
VHRplus (or VHR+) = Very High Resolution (increased quality compared to VHR)
YFS = Young Farmers Scheme
TERMINOLOGY
Beneficiary: as referred to in Article 2(1) of Regulation (EU) No 640/2014;
Control population = beneficiaries applying for an area-related aid scheme;
Control population for greening = beneficiaries required to observe the
greening practices and who are not exempted or who are not participating in a
certification scheme;
Control sample = sample of beneficiaries selected for an on-the-spot check;
5
Established area: Area of EFA resulting from direct field measurement or from
delineation using ortho-imagery;
Greening payment = the payment for agricultural practices beneficial for the
climate and the environment as referred to in Chapter 3 of title III of Regulation
(EU) No 1307/2013;
Random sample = group of beneficiaries selected randomly;
Risk-based sample = group of beneficiaries selected on the basis of a risk
analysis.
2 Overview
The overall process of OTSC is provided in figure 1 following a chronological
approach. The different steps will constitute the chapters of the current document.
A general timing and responsibilities for the different steps of CwRS campaign is
also provided in table 1.
Member State administrations have to define their OTSC strategies and methods
in such a way as to ensure effective verification of the correctness and
completeness of the information provided in the aid applications and to ensure
compliance with eligibility criteria and other obligations (Article 24 of Regulation
(EU) No 809/2014). To do so, administrations are advised to reflect, not being an
exhaustive list, the geophysical particularities of the country, the different farm
topologies, the staff and budget availability, the topologies of the different
payment schemes, etc.
Farmers are required to submit their annual subsidy applications in prescribed
form and by dates set in line with the Regulation. A sample of the whole population
concerned by each payment scheme has to be controlled On-The-Spot. The check
of the selected applications can be based on farm visit, or on satellite or aerial
remote sensing data, or on a combination of these methods. Part of the OTSC
work can be done through the use of external contractors.
Prior to the actual performance of checks, it is necessary to ensure a crucial step
of data and tools collection and preparation.
Then the real work of OTSC of selected applications will take place. When using
Ortho imagery (CwRS), each agricultural parcel will be categorized separately by
applying the decision tables established by the Administrations to reflect the
diagnosis concerning the area, the land use, the GAEC, the Rural Development
measures according to the decisions of the Member State.
The photo-interpretation of agricultural parcels will normally be carried out using
at least one very high resolution (VHR) image (aerial orthophoto or satellite ortho-
image with a pixel size <=0.75m, (<=0.50m if dealing with small Landscape
features)) of the current year to allow the area check of agricultural parcels. Their
land use, wherever necessary, will be checked with the addition of a VHR image
6
or multitemporal high resolution (HR) images. The latter can be Sentinel2, HR or
HHR.Sentinel1 data (Radar) can also be used. Attention should be paid to the fact
that Radar data processing requires skills and knowledge different from optical.
Please refer to the dedicated chapter in the technical guidance for OTS check of
Crop Diversification).
In the case where the diagnosis may not be completed by computer-aided photo-
interpretation (CAPI) procedures alone, "rapid field visits" (RFV) will be carried out
by the contractor or the Administration for checking the land use and/or some
other issues.
Administrations may also opt for the use of only one VHR image to check areas
and perform a field visit to check the land use and/or some other issues on all
selected parcels (so-called: Systematic Rapid field visit).
Following the parcel checks, dossiers will be categorised at payment group level
and then at dossier level. Unless specified otherwise, the contractor will participate
only in the stages related to the analysis of the remote sensing imagery
completing the diagnosis. The penalty calculations, sanctions or financial
consequences for the farmer are the responsibility of the Administration.
For each dossier inspected OTS, a report has to be produced to document all steps,
findings and results. Attention should be paid to any finding that could trigger an
LPIS update.
At the end of each OTSC campaign, it is highly recommended to perform a work
of analysis of the results of the campaign. That consists in doing: a Quality Check
(internal and/or external) of OTSC work (both field and CwRS) by re-performing
the checks of a set of dossiers; an analysis of the error rates obtained and the
main causes that led to these errors. Lessons learned should be capitalised to fine-
tune, update if not upgrade OTSC methods for the following OTSC campaign.
The Commission’s contribution to the OTSC programme is restricted to the
technical coordination of methodological choices and the provision of satellite
imagery for control zones defined by the Member State. Since 2007, the quality
control procedure is the responsibility of the Member States. However, since 2016,
the JRC offers the possibility to Member States to perform a QC of the CwRS work
on one (or two) CwRS zone(s).
7
Figurer 1: Overview of the OTS checks processes
8
Table 1: Main stages of a CwRS campaign
Responsible Description Period
Preliminary work
Administration
MS
Selection of control zones, assessment of image
requirements (interfacing within the web based
system G4CAP)
Publication of Technical Recommendations
September-
January
Administration
MS
Call for tenders, selection of contractors, signature
of contracts
December- March
Commission Updating of technical documentation in WikiCAP All year long
Administration
MS
Selection and administrative processing of
applications lodged in the control zones; transfer to
contractors of dossiers and data bases
(declarations, digital LPIS and LPIS ortho-images)
April- June
Contractor Collection of applications data, LPIS vectors and
farmer’s sketch maps
March- June
Preparation of data
Commission/
Contractor
Acquisition of satellite images (Commission) and/or
aerial photographs (Administrations/Contractors),
processing, geometrical correction etc.
From October
until the end of
the campaign
Photo-interpretation and categorisation of
applications
Contractor Photo-interpretation of parcels to be checked May- August
Contractor Categorisation of applications and return of results June- August
Administrative follow up
Administration
MS
Follow up inspections in the field (if included in
CwRS strategy)
July- October
Contractor Contractor's report to Administration and
discussions of results; return of summary statistics
and images (raw and rectified) to the Commission;
October-
December
Commission Possible visits to contractor and Administration on
request by a MS
January –
September
3 Documentation, justification and decisions sharing
The implementation of the CAP legislation is almost entirely the full responsibility
of Member State administrations. Indeed, following the last CAP reform, more
responsibility than ever has been given to Member States for the definition and/or
9
decision of many components of the CAP system (e.g. minimum farming activity,
list of EFAs, list of GAECs, and definition of a tree …).
When a Member State is audited (DG AGRI audit services, European Court of
Auditors, etc.) the main aim of auditor work is to check that the management and
control system set by the Member States is reliable and effective (i.e. ensuring a
minimum level of residual error and at the same time meeting the policy
objectives). Since different solutions may be considered to set a method, and
many elements of the IACS may lead to weaknesses in the system, it is
recommended to administrations to document, justify and provide evidence as to
their choices, decisions and actions.
It is also very crucial that procedures decided and decisions and rules set are
largely and openly shared between all stakeholders of the IACS (Administration,
controllers, farmers, and Commission on request). Commission guidance,
technical guidance and WikiCAP webpages can obviously be used for that purpose.
However, it is again recommended to develop and make use in each Member
States of solutions like:
- Dedicated Website providing all information on the IACS;
- Webpages dedicated to FAQ (frequent Asked Questions);
- Production of thematic Leaflets or booklets (the application system, the
different payment schemes, the EFA list, definition and rules, The GAEC
list definition and rules …);
- Production of “Controller guide”;
- Organisation of trainings and/or information days;
- Use of Farm Advisory Service.
4 Preliminary work
3.1 Define strategy
Member State administrations have the entire responsibility and a large autonomy
to define their OTSC strategy. Strategies and methods should be set with the sole
aim to ensure effective management and verification of the correctness and
completeness of the information provided in the farmers’ aid applications.
Plausibly, the definition of sound strategies requires a good knowledge and
analysis of the Member State and/or Regions farm structure and farmland
characteristics. To do so, administrations should make use of GIS technologies.
Different layers and data set (e.g. LPIS layer, EFA layer, digital applications, road
network layer …) could be combine and queried to help optimising field checks
organisation, defining CwRS strategies, performing risk analysis, and optimising
CwRS imagery use.
In addition, administrations can make use of the different results and findings of
Quality Checks (see dedicated chapter), the LPIS Quality Assessment (see LPIS
QA WikiCAP pages), or even the residual errors to adapt and fine-tune their control
strategies.
10
3.2 Sample selection
The system “in cascade”, entered into force on the 1st of January 2015, of selection
of the samples to be OTS checked for the different payment schemes (BPS/SAPS,
Greening, VCS, SFS ...) and policy measures (Cross Compliance, Rural
Development measures …) is described in guidance DSCG/2016/32 Final- Rev 2
final. Selections are now mainly random based. Some elements (e.g. ‘Greening’,
Cross compliance …) also include a risk based part.
A. Random selection
- The random sample concept
The random sample permits an estimate of the background level of anomalies in
the system. It supports decisions enacting the mechanism for increasing the
control rate (in accordance with Art.35 of Regulation (EU) No 809/2014) and also
permits an assessment of the effectiveness of the criteria being applied for risk
analysis.
There are several types of random samples, as well as different methods for
compiling the random samples. For more details, please refer to section 1.3.2 of
document DSCG/2016/32 Final- Rev 2 final.
B. Risk analysis and annual assessment
According to Art. 34(5) of Regulation (EU) No 809/2014, MS are responsible for
the definition of the risk criteria to be used for the risk analysis. It is the MS'
responsibility to assess the effectiveness of the risk analysis on an annual basis
and to update it by establishing the relevance of each risk factor. For further
details, refer to section 1.4 of DSCG/2016/32 Final- Rev 2 final.
The risk factors can be qualitative (e.g. region/district/department …) or
quantitative (e.g. area declared, ratio of crops, EFA declared, number of
participating schemes…). It is worth noting that the risk factors might be deferent
among schemes and/or among the MS. The only constraint is that each risk factor
must be known for each beneficiary in order to assess his own risk.
Depending on the scheme, several variables are influencing both the reductions
and the penalties. However, for area related subsidies and for ‘greening’
measures, the ratio of “area not found” (RAnF),) i.e. the total area not determined
in the relevant crop group/EFA over the total declared area for the same crop
group/EFA, is a common parameter. For the CD in the ‘greening’, the other
parameters are the declared shares of each crop within declared AL.
For this, MS can rely on a CART model (i.e. Classification and Regression Tree)
with the area not found in individual claims as the dependent variable (i.e. the
variable to be predicted). The aim of the CART model is to rely on a set of
independent variables (i.e. the explaining variables; here, the potential risk
11
factors) in order to find homogeneous sub-groups of the population (called the
“nodes”). Advantages of the CART model are that it is:
- Well implemented in various statistical software (e.g. Matlab, R, S+,…)
- Relatively easy to apply (it only requires the input of the dependent variable
and the potential risk factors)
- Flexible (no assumption is made how the potential risk factors are affecting
the dependent variable)
- Irrelevant risk factors are automatically excluded from the model
When calibrating the model, attention must be paid to the maximum level of the
tree (i.e. maximum number of consecutive nodes) and the minimum number of
observations in a node (generally at least 50).
After calibrating the model, a procedure named “pruning” must be applied in order
to remove the insignificant nodes in the tree. Ideally, the procedure is sequentially
repeated to get simpler and simpler models. The final model is then chosen by
optimizing criteria (e.g. minimum predicted variance on the validation set). If
possible, the validation set should be independent from the calibration set (i.e.
the individual claims that were used for the calibration should not be used for the
validation).
Using the final CART model, it is possible to estimate the RAnF for each application
and thus to estimate the discrepancy between the claimed area and the paid area
(i.e. after deduction of the reductions and the penalties). These estimations on
the whole population of beneficiaries can be used within the following alternative
procedures:
- set a threshold determining the “low” and “high” risk sub-population, then
select randomly within the “high” risk sub-population;
- as proxy for a probability-proportional-to-size sampling of the applications
(ensuring thus to sample mainly the larger expected errors;
- regroup the applications with similar estimated risk (e.g. classes of
discrepancy between claim and payment). A stratified random sampling can then
be applied on these strata with a sample rate per stratum determined by the total
risk of the corresponding stratum.
For instance, in the third alternative, a risk stratum that covers 30% of the total
risk of the population (i.e. the sum of the estimated risk within this stratum is
equal to 30% of the total sum of estimated risk) should represent 30% of the total
sample size even if they are composed of only 10% of the total population.
Knowing this sample size for the stratum, we can then translate it into a sample
rate for the stratum. Thus, if the total population is 100k claims, the risk stratum
above should be sampled as follows:
12
# of
claims
(a)
% of
claims
(b)
Estim. Risk in ha
for the strat (c)
Share of estim.
Risk for the
strat (d)
# of
sample
(e)
Sample
rate
within
strat (f)
Highest
risk
5k 5% 400 ha 40% 1.6k 32%
High risk 10k 10% 300 ha 30% 1.2k 12%
Medium
risk
30k 30% 200 ha 20% 0.8k 2.7%
Low risk 55k 55% 100 ha 10% 0.4k 0.7%
TOTAL 100k 100% 1000 ha 100% 4k 4%
In the example, column (e) is computed as 4% x 100k x (d) (e.g. the first row is
4% x 100k x 40% = 1.6k), 4% being the objective risk-based sample rate. That
is the number of claims that must be selected in the stratum and put in the risk-
based sample while column (f) is computed as (e)/(a) (e.g., for the first row is
1.6k/5k = 32%) and is the percentage of claims within this stratum that must be
selected.
Comparison of the three suggested risk-based sample selection
Risk-based
selection
method
Pros Cons
Random
selection
within the
“high risk”
class only
Optimal if the RA
is perfect
Easy to implement
High impact of
the quality of the
RA
Requires the
choice of the
threshold
between “low”
and “high” risks
Proportion-
to-size
sampling
Chances of
selection
proportional to its
own risk
Error rates might
be less marked
compared to
those of the
random sample
13
Comparison of the three suggested risk-based sample selection
Each beneficiary
has chances to be
selected
Lesser impact of
the quality of the
RA
Stratification
of risk levels
In principles,
targets more the
sub-populations
representing the
largest shares of
risk
Lesser impact of
the quality of the
RA
Requires to define
the number of
classes
Requires
thresholds for the
different classes
Error rates might
be less marked
compared to
those of the
random sample
4.3 Selection of Control zones
Contrary to classical checks which can be individually geographically dispersed, in
the case of CwRS, the areas where imagery is to be acquired need to be
established. This clustering of checks is called a "control zone", and is a
geographical area defined on the basis of GIS analysis). It is worth noting that
classical checks can also be organized in geographical clusters. The following
descriptions can thus be arranged for any combination of classical checks and/or
CwRS.
Despite the clusters, it is essential to ensure the representativeness of all
conditions/ requirements of the targeted schemes and measures in the choice of
the RS zones (see in particular Art. 34(2) last sub-paragraph of Regulation (EU)
No 809/2014). This is particularly true as RA is mainly foreseen to be used for the
greening samples. The Commission services hence recommend opting for a higher
number of zones of small size.
It is reminded that the location of the zones shall remain confidential. In case of
the use of contractor, zones should not be disclosed until a contract has been
awarded.
The RS zone can be selected randomly and on the basis on a risk analysis. The
selection possibilities are detailed in section 1.5 of document DSCG/2014/32 -
FINAL REV 2 Final.
- For the Selection of dossiers
14
For the selection of dossiers, and especially for zones to be checked with a VHR
satellite image to be acquired on Commission budget, attention as to be paid to
the cost/benefit of such imagery. For instance, even if no thresholds are currently
imposed by the Commission, it is recommended to look for the optimisation of
parameters like the total area of the sample of applications to be checked inside
the total Area of Interest (AoI) of VHR image.
- In case of contractor
After the phase of samples selection, approximate figures on the number of
dossiers and zones should be given in the National Addendum. The approximate
size of the zones should also be given by the Administration to the bidders. These
figures, that should be taken as provisional, are intended to help the bidders
assessing their workload. Also, in order to help the bidders determining a mean
cost per application, the Administration should indicate the mean number of
parcels per application and per zone, if requested by the bidders, as this number
may vary significantly between regions of a given Member State.
4.4 Suggestion of an iterative procedure for selection of control zones and samples
DG JRC suggests to rely on the following procedure the selection of both the
control zones and the different OTS samples (random and risk). The procedure is
in line with the “cascade” sampling) and can be applied for any combination of
classical checks and/or CwRS. In the description of the procedure, it is assumed
that the RAs have already been conducted so that the risk associated to each
individual beneficiary can be evaluated. The description is made for the BPS/SAPS
population but can be translated to other schemes (e.g. second pillar).
Step 1: Anticipatively estimate the required number of controls for each
considered schemes accordingly with the beneficiary population. For instance, the
share of the initial random selection is 20% (1% out of 5% of the controls), while
the expected share (noted Sg% hereafter) of the ‘greening’ risk-based sample is
the ratio between the number of risk-based ‘greening’ sample on the total number
of controls (i.e. 5% of the population);
Step 2: Select randomly a control zone;
Step 3: Apply the “cascade” sampling on the zone using the expected shares
computed in Step 1. In principle, all the beneficiaries of the control zone could be
selected for controls in any of the different schemes. By doing so, the cost
efficiency is guaranteed. However, this is not mandatory and the beneficiaries that
remain unselected can be put in a reserve list for completing the sample at a latest
stage;
Step 4: Add the different samples to the samples already selected on the previous
control zones and check whether the target sample sizes are reached;
Step 5: If all the target sample sizes are reached, stop the procedure.
Otherwise, repeat Steps 2 to 4 until all sampling targets are met.
In addition, it is recommended to:
15
- Consider to pre-select a slightly larger sample in order to guarantee that
the target sample size will be reached;
- Make the pre-selection estimation on slightly smaller zones than the final
zones to increase the chances that the beneficiaries can be controlled (especially
when using CwRS);
- Consider the possibility to switch from CwRS to classical checks if the
control zone covers only few beneficiaries;
- Consider to select the last control zones in accordance with your needs. For
instance, if the risk-based ‘greening’ sample is under represented, you might want
to select the last control zone(s) by RA.
5 Preparation of checks
In order to check the conditions under which an aid is granted it is crucial to ensure
a comprehensive step of data and tools gathering, pre-processing and processing.
5.1 Reception of applications and data entry
From 2018 onwards, all farmers will have to provide information for their aid
applications and payment claims, via a GIS-based interface (the geo-spatial aid
application - GSAA). See guidance document on aid applications DSCG/2014/39
FINAL-Rev 1.
The implementation of GSAA in Member States is phased-in from 2016 onwards
(cf. Art. 17 of Regulation (EU) No 809/2014). So, in 2015 aid applications can still
be provided in paper format. In view of facilitating the submission by the
beneficiaries and to reduce the risk of errors, Member States shall provide pre-
established forms (digital or paper format). Through these forms, it is advised to
provide as much information as considered useful based on diagnoses made on
parcels in the previous year as well as the most recent graphic material indicating
the location of those areas.
Furthermore, to limit errors, it is recommended to avoid to require hand written
or free text information. Farmers should provide information via drop-down menus
or lists with tick boxes using pre-defined common options and nomenclatures.
If an administration make use of a contractor, ideally, the applications will be
transmitted to the contractors in digital form, after having been subjected to
consistency checks possibly under an anonymous form. The format of the
database given to the contractor will be described by the Administration, and
accompanied by a list of the codes to be used.
5.2 Collection of LPIS data and other ancillary documents
Access has to be prepared and made available to the controllers for the relevant
parts of the LPIS and EFA layers (i.e. the reference vectors, as well as the
reference areas) and if any, the associated orthoimages. The applications to be
16
controlled should contain appropriate cartographic documents allowing to locate
all agricultural parcels, and, if applicable, all required EFA elements on the ortho-
images or inside reference parcels.
5.3 Provision of satellite images
Since 1993, DG AGRI has promoted the use of “Controls with Remote Sensing”
(CwRS) as an appropriate control system suitable to checking if aids are correctly
granted. On the basis of the Council Regulation (EC) 1306/2013 and of the
Commission Implementation Regulation (EC) 809/2014, 908/2014, the
Commission Services are required to centralize the Satellite Remote Sensing
(SRS) image acquisition. This task was transferred to DG JRC in 1998 (September
1998/VI/34942) and it is managed through a horizontal co-delegation (Type I)
between DG AGRI/DG JRC (via DG BUDG) to implement the yearly CAP image
acquisition work programme.
Regards to timing of the operations the Commission Implementing Regulation
(EU) No 908/2014, mentioned above, its art 26, says:
1.) For the purposes of Article 21 of Regulation (EU) No 1306/2013, each
Member State shall inform the Commission by 1 November of each year at
the latest, as to: (a) whether it wishes the Commission to acquire the
satellite images necessary for its programme of checks and/or for its Land
Parcel Identification System Quality Assessment; (b) the area to be checked
and the number of planned control zones.
2.) Member States requesting the Commission to obtain the satellite images
shall finalise, in cooperation with the latter and before 15 January following
the communication of information referred to paragraph 1, the zones to be
covered and the timetable for obtaining those images
With reference to Table 1 of Chapter 1, MS Administrations shall insert at first
their pre-Image Requirements (pre-IR), and then the final detailed image requests
in the web application G4CAP in accordance with above deadlines.
For details on the web application G4CAP used to synchronise all stakeholders in
the satellite image acquisitions for the CAP please refer to:
https://g4cap.jrc.ec.europa.eu/g4cap/
1.) Manuals are found under ‘Documentation’ menu (no credentials needed)
2.) For all interaction user/pwd (credentials) will be requested from JRC.
5.4 Management of satellite image acquisition
All instructions valid for the stakeholders (or Actors) participating in the CAP image
acquisition: DG AGRI, JRC, MS Administrations, MS Administrations contractors,
FW contractor/s (with their two roles: operator and image provider) are given in
the ‘image specifications for the CAP checks’ which are updated every campaign.
These specifications are divided into the VHR, VHRplus (or VHR+, of enhanced
quality) specifications, and the HR and HHR (of enhanced quality) specifications.
17
Detailed specifications of each type of imagery is available on the G4CAP
website.
The macro actions in the image acquisition process described in detail in above
specifications follow the below workflow. To the right is the responsible actor.
Figure 2 – the satellite image acquisition process with relative responsible stakeholders
A) Use of Copernicus datasets The MS Administrations and their contractors should be aware of the possibility to
use the Copernicus infrastructure sensors Sentinel1 and Sentinel2, and in addition
its DataWareHouse (DWH) datasets.
The Sentinel datasets are free of charge and openly accessible2. The EC suggests
to use all cloud free S2 pixels wherever useful in their CAP controls. Sentinel 2A
repeat cycle at equator is 10 days, which means approximately 35 passes/year.
Due to overlapping swaths from adjacent orbits in “mid-EU” there are some 40-
50 passes/year, i.e. approximately one pass every week i.e. 6 passes per ‘normal’
2 S1/S2 accessible freely from the ESA scientific hub: https://scihub.copernicus.eu/
18
length CwRS window (with one satellite). When Sentinel 2B is launched (late 2016,
early2017) passes will double.
It is up to the MS Administrations to download S2 data from ESA or elsewhere.
However the JRC makes available the ‘S2 alert module’ accessible from within the
G4CAP web application where the user can be alerted and given the access URL
whenever there is a suitable S2 product/granule acquired over a defined control
zone over a defined window.
The DWH datasets follow a licensing agreement that may allow download (DL), or
VIEW, freely for Public Authorities3, after registration to the Copernicus Space
Component Data Access (CSCDA). These DWH datasets include a vast number of
CORE and ADDITIONAL datasets, including pan EU HR and VHR coverages which
are renewed every 3 years, and can be used most suitably in preparatory work
for the CAP controls.
5.5 Acquisition of aerial imagery (if applicable)
Acquisition of aerial imagery for the controls is the responsibility of the MS
Administration, i.e. it is not coordinated by the Commission. The main advantage
of aerial imagery with respect to VHR satellite imagery is that it allows covering
much larger areas (e.g. large administrative units such as full provinces) in a
limited period of time. Alternatively, a large number of small zones may also be
covered in a given region.
However, acquiring aerial imagery has also some proper constraints such as
restrictions over military zones and air traffic lanes. Cloud cover is not as
restricting for aerial photography as for satellite imagery, but meteorological
conditions are in any case affecting the radiometric quality of the images.
Moreover, the lead-time in the acquisition-processing of aerial imagery may be
longer than that for satellite images. Aerial images acquisition must therefore be
organised sufficiently in advance, and the acquisition periods should be relatively
early in the year. The use of (natural colour in combination false colour composite)
imagery permits an easier identification of land covers, thus significantly reducing
follow-up rapid field visits for crop identification.
It is also compulsory that aerial flights be carried out within the present state of
the art: the use of GPS and inertial navigation systems linked to the camera makes
it possible to optimise the flight coverage and considerably reduce the costs of
further processing.
If Administrations and/or contractors intend to acquire and/or ortho-rectify aerial
digital imagery, they should refer to the Best Practice and Quality Checking of
Ortho Imagery JRC guidelines4.
3 DWH (CORE datasets …) of the Copernicus Space Component Data Access (CSCDA):
https://spacedata.copernicus.eu/ 4 https://g4cap.jrc.ec.europa.eu/g4cap/ (under Documentations menu)
19
5.6 Processing of satellite images and aerial photographs
Note (1) that, "image" will refer both to the satellite and aerial images.
Note (2) that the person in charge of image processing should record and justify
all steps and all processing techniques used: geometric ortho correction,
radiometric correction, contrast stretching, resampling, pan sharpening,
mosaicking.
A. Geometric correction The Commission does not impose a methodology for geometric correction of
imagery, but gives recommendations and guidelines in line with its Quality
Assurance (QA) strategy described in the Guidelines for Best Practice and Quality
Checking of Ortho Imagery. The philosophy is to have a set of suitable control
procedures to ensure a satisfactory quality of the product. These guidelines are to
be considered as the Commission’s current understanding of “best-practice”. It
must be clear, however, that in the end the administration/contractor alone is
responsible for the accuracy of products.
The entity in charge of image processing needs to have appropriate software
suite (or sub-contracting options), and “know how” to process all image types
(i.e. be able to ortho-rectify all image types, pan-sharpen bundle images, etc.).
The allowed geometric error in the output images (and associated Digital Elevation
Models, DEM) are expressed as a maximum permissible absolute (i.e. with respect
to a specific geodetic reference frame) Root Mean Square Error (RMSE) with
respect to check points, and are stated in the Technical Specifications mentioned
above. The geometrically corrected products and associated DEMs are assessed
separately in up to three geometric dimensions RMSEx, RMSEy, and (where
relevant) RMSEz.
Since 2014, the EC has introduced the use of so-called VHR and HR image profiles.
Examples are VHR prime, VHR topographic, VHR LPIS (special case), VHR pan,
VHR stereo, VHR archive, and VHR backup. These are sensor independent profiles,
fitting the CAP VHR and HR image specifications in terms of spatial resolution,
radiometric resolution, number of spectral bands, geolocation accuracy/absolute
geometric accuracy achievable in orthorectification, elevation angle, programming
type, processing levels, and cloud cover threshold over the control zone (AOI) etc.
These profiles are explained in the specifications where orthorectification
thresholds allowed are given for all sensors included in the different profiles (i.e.
1-dimensional RMSE ≤ 1.25m etc.). It is essential that the MS Administrations
and their contractor keep within given thresholds in orthorectification. Below
figure shows a summary of the VHR and the HR profiles.
20
Fig 3 – VHR/VHR+ profiles adopted within the CAP checks
21
Fig 4 – HR/HHR profiles adopted within the CAP checks
In order to meet these specifications, the person in charge should carefully analyse
the input data and particularly the digital Elevation model (DEM) to be used, the
ground reference data (accuracy of Ground Control Points (GCP), independent
checkpoints (ICP) used, their source, number and distribution) and each step of
the geometric correction process. It is advised to detail and record all steps of this
geometric correction process and justify the correction method used (e.g. ortho-
correction or polynomial) for the control zones.
B. Radiometric correction Imagery checks, especially for the LPIS QA screening, revealed that the quality
measures related to radiometry and photometry given in the JRC orthoguidelines
and specification for orthoimagery needed some further clarification. Details are
provided in Annex 1.
5.7 Ground data collection
As example for the CAPI and/or training for classification of the satellite images,
it is recommended to build a database of reference fields for the most common
22
crops in control zones. The survey(s) have to be carried out at the beginning of
period(s) most appropriate for the crops of interest (winter crops, summer crops,
secondary crops, permanent crops …). Surveys should ensure a good
representation of the main crops (5 to 10 parcels per crop type spread over control
zones) and include, if applicable, some examples of rare crops.
5.8 Production of controller guide (CAPI guide and field control)
One of the main elements influencing the accuracy of On-The-Spot checks results
is the use of appropriate tools and the use of tools appropriately. Of great
importance are the defined rules to identify, delineate and measure features and
crops of interest for the different CAP schemes.
The production of controller guide is thus crucial to document and share the
defined rules among the different stakeholders.
In this guide, attention should be paid to the provision of clear feature specific
identification rules applicable both on imagery and on the field. The basic
assumption is that a specific feature on the ground remains obviously the same
feature on imagery; thus, its delineation and/or measurement should come with
the same result whether it is checked using imagery or in the field.
The guide should contain drawing and example pictures to illustrate specific cases,
provide information on possible ancillary data (archive imagery, river, transport
networks etc.) that could help identifying and delineating features.
5.9 GNSS and ortho image validation
The use of tools - whether ground or remote sensing based - for the measurement
of agricultural parcel areas requires that all interested parties can expect reliable
and trustable measurements.
The JRC has developed and proposed a method for the validation of tools for area
measurement, which can objectively be shown to perform correctly under specific
conditions, and considered therefore fit for the purpose of area measurements
made in the context of field checks required in Commission Regulation.
While not mandatory, it is highly recommended to Member States to validate their
measurement tools through the JRC method. Alternatively, Member States can
also:
- use a device tested and certified by an authorized entity;
- use a device tested and validated by a reference laboratory approved by
JRC.
The full description of the JRC "Area measurement tool validation method" is
provided in Annex 2.
23
6 Perform checks
The purpose of on-the-spot checks is to check the conditions under which aid is
granted in accordance with Article 37 of Regulation (EU) No 809/2014. In
summary, this means to check for each parcel declared: their location, area
related aspects and land use/land cover related aspects. Diagnoses will be
attributed at parcel level.
When using imagery, in case of non-conclusive opinion on a parcel, a rapid field
visit shall be undertaken for completion of the diagnosis at parcel level.
Then, diagnoses at parcels level will be aggregated to provide a diagnosis at crop
group and/or payment scheme level. Finally, a diagnosis will be provided at
dossier level.
As a by-product of these checks, feedbacks to the LPIS should be made (e.g. using
specific codes) wherever appropriate.
6.1 Check at parcel level
A. Parcel location
In the field, documents such as maps for the overall location of the parcels and
detailed location documents (e.g. parcel boundaries overlaid on a VHR image) will
have to be provided to the staff in charge of the checks.
Alternatively, navigation systems based on GPS and systems allowing the display
of images, vectors and data on a mobile computer in the field may be used to
locate and reach the parcel of interest.
On imagery, each declared parcel will be located on screen with the help of the
reference parcels (LPIS) vectors and the farmer’s sketch map wherever
available. It is reminded that through the Geo Spatial Aid Application (GSAA) and
also on the paper declaration, the farmer shall indicate the location of each
agricultural parcel on the graphical material supplied to him by the Administration.
It is important to locate and delineate all declared parcels, including those for
which no aid is claimed, so as to detect possible multiple claims and to verify cross
compliance issues.
B. Area check The inspector should have received sufficient instructions and training, and be
largely able to undertake the work autonomously. The inspector should have no
conflicts of interest, and should be able to carry out the inspection independently.
In order to provide a result to the appropriate precision and to ensure effective
verification, the inspector must have access to appropriate claim data (including
map information) and measuring equipment.
Also, as a general rule, the area of each subsidised agricultural parcel will be
verified either on the field or on the current year VHR imagery (i.e. <= 0.75m
pixel size). Unless requested otherwise by the Administration, the area of non-
24
subsidised agricultural parcels will, in general, not be checked. The result of the
digitization or field measurement will be the “measured” area, which will be
compared to the declared area for each agricultural parcel. The results will be
expressed in hectares rounded to two decimal places. Also, the area measured
will be expressed as the area projected in the national system used for the LPIS.
Attention should be paid to the definition of the agricultural parcel attached to the
crop group or payment scheme under check. The inspector/controller shall follow
this definition when measuring the claimed parcels (see ‘definition of agricultural
parcel’ chapter of DSCG/2014/32 Final- Rev2 final).
The total area of the agricultural parcel should be measured. However, areas not
taken up by agricultural activities such as buildings, woods, rocks, some ponds
and paths are to be excluded by deduction from this area.
To assess the eligibility of eligible area within an agricultural parcel of permanent
grasslands, Member States can use a reduction coefficient (please refer to the
Technical guidance on the pro rata system for permanent grassland).
Member States shall define beforehand the criteria and procedure used to delimit
the (in)eligible part of the parcel in order to ensure that these criteria are
communicated to farmers, where necessary, correctly transposed in the LPIS and
adequately included in the instructions for the on-the-spot checks.
In the office, on-the-spot checks need to be prepared by selecting the areas to be
measured, grouping and/or subdividing the parcels declared with reference to the
LPIS, and arranging visit itineraries as efficiently as possible so as to cover one or
more holdings. The direct – preferably in digital form – consultation of the LPIS
data (including where applicable orthophotos) shall be possible for the preparation
of field controls. The LPIS data should be available also for use during the on-the-
spot checks.
Where features that are part of the good agricultural and environmental condition
obligations or the statutory management requirements (e.g. hedges, drainage
ditches, small woods according to the local regulations) have been specifically
recognised and defined as (landscape) features eligible for area payment, they
should not be excluded from area measurement. Nevertheless, it is recommended
that, during the on-the-spot checks (i.e. remote sensing or otherwise), such
features should be digitized as points, lines or polygons with their corresponding
attributes in the LPIS, this way making possible the control of their maintenance.
There are two options for measuring the agricultural parcel area: 1) direct
measurement or 2) measurement by deduction, which is applicable only in
particular circumstances.
The measurement by deduction can be done when the Land Parcel Identification
System (LPIS), together or not with other ancillary data such as ortho-photos,
permits the confirmation of the boundaries of the declared agricultural area. Thus,
the area measurement may focus on the determination of ineligible areas and
deductions. This situation is only possible when:
- the LPIS reference parcel is an agricultural parcel; or,
25
- the reference parcel is fully declared; or
- use is made of geospatial declaration (or graphical material which is then
transcribed in the GSAA in accordance with Article 17(3)(b) of Regulation
809/2014) of agricultural parcels, which allows an overlay of boundaries and
eligible area as reported on the image;
- and areas not to be accounted for can be easily identified.
In all other circumstances, a direct measurement of the parcel area (e.g. using
GNSS or VHR ortho-image) is required.
For details on the deduction of ineligible features, please refer to section 3.2 of
document DSCG/2014/32FINAL- Rev2 final.
C. Material to be used for parcel area measurements MS shall use measurement tools that are proven to assure measurement of quality
at least equivalent to that required by applicable technical standard, as drawn up
at Community level. In other words, accuracy of measurements must be at least
equivalent to the accuracy needed for the management of geo referenced data at
1/5.000 scale. The quality of a given tool is defined by the tolerance (i.e. buffer
width) applicable to this tool as determined through an area measurement
validation test. Only tools (e.g. GNSS equipment, remote sensing ortho-
images) with a buffer width not exceeding 1 m should be used.
- Use of VHR ortho-imagery
Current year VHR imagery (i.e. pixel size <=0.75m) should be used for parcel
area measurement. Recent archive VHR imagery can be used only if it enhances
the interpretation of the current year VHR imagery, therefore helping the
interpreter making the decision. It is reminded that VHR imagery should be
accurately ortho-rectified (see corresponding section). For the delineation of
parcel boundaries, a viewing scale of 1/1.000 would often be suggested. However,
the interpreter should vary the scale depending on size and type of parcel to be
measured and store this information in the OTSC report.
Where reference parcels (LPIS parcels) contain several (full or partial) agricultural
parcels, the CAPI operator will have to locate and digitize the declared agricultural
parcels inside the LPIS parcels using the sketch maps attached to the farmers
application and taking account of the current definition of the agricultural parcel.
Since farmers’ sketch maps are only indicative, operators are advised to report
cases where the retained area significantly exceeds the declared area so that
complementary checks may be carried out by the Administration (particularly for
dossiers where the possible excess retained area compensates for an over-
declaration). Such cases may occur when the operator is not aware of all declared
parcels in the reference parcel (e.g. as a result of the sample selection) or because
some parcels may not be declared (e.g. because they belong to non farmers).
- Use of Stand alone GNSS devices
The GNSS equipment offered on the market is very wide, but not all devices meet the accuracy requirements imposed by the Regulation. Nowadays, most of devices come with dedicated packages allowing measuring parcel areas. If an Administration decides to acquire a new GNSS receiver, it is advised to perform an
26
area measurement validation tests need to know if it meets the required accuracy.
How to measure the area? The general method to measure a parcel area consists in following the parcel perimeter with the GNSS device. It is reminded that it is the GNSS antenna that should always be positioned and maintained at the vertical of the perimeter to be measured. The user should be aware of the possible difficulties that may arise during measurement, such as loss of satellite signal, multi-path effect etc. Some of these difficulties can be reduced by an appropriate set up of parameters like: signal to noise ratio (S/N), maximum Position Dilution of Precision (PDOP) and horizon mask. In any case, a visual check of the parcel recorded should be systematically be done to detect blunder points (see Figure below).
Figure . Example of an error connected with loss of satellite signal.
More generally, large discrepancies with the declared area should, in case of doubt, always lead to a second measurement. Whenever the measurements need to be taken in a difficult area like a valley or neighbourhood of a forest, a measurement planning software is advised to be used. This software allows simulating the configuration of the GPS system at a certain point and time of the day, month, year. As the position of satellites is changing in time, selecting the optimum time of the day for the measurement can help achieving a reliable result in a short time.
27
Figure left: Simulation of number of satellites in the certain position with some
obstacles on the side. Figure right. Simulation of the Horizontal Dilution Of Precision (HDOP) in the
certain position with some obstacles on the side.
Some software are also able to take into account the features potentially blocking the signal from the satellites. This is done by introducing a simple sketch of the position of the obstructions influencing the test field in the software. The horizon is therefore reduced according to this sketch, making the simulation more realistic (see Figure 4.).
Figure: A sketch of the situation in the field. A feature blocking signal on the
west side of the object to measure (in grey).
Continuous measurements or logging vertices only? The appropriate method of measurement as well as advice for optimizing the measurement accuracy are usually suggested by the manufacturer. Nevertheless,
as a general rule, in open horizon, the use of the continuous mode is recommended as it increases the possible compensations between point position
errors. However, where obstacles are present (e.g. a wood or a hedge), the stop & go method may give more precise results. Please note, that the result of the validation is strictly related to the tested method
of measurement and not only to the device. Therefore, it is recommended to perform the GNSS are validation test in both continuous and vertex methods.
For parcels with very linear borders, one can decide to perform measurements with logging vertices (at each corner) method. However, it often proved to be
significantly longer than the continuous measurement method, since several logging vertices must be recorded.
An often commented fact is that the tracks of the measurements taken with the continuous measurements method look 'worse' (more noisy) on the screen of the device (and in the GIS) than the ones collected by logging vertices only. The
28
purpose of the on-the-spot controls is to find the actual area eligible for payments and to verify the farmer's declaration in a fair way. Therefore, the reliability of the
measurement and the best practice in taking the measurement should be a priority over the 'sharp' shape of the field. In other words, the method of the
measurements should be adjusted to the tool and conditions of the measurements rather than to the preferences for seeing 'straight' borders in the GIS database.
- Use of Differential GPS, geodetic survey GPS or Total station
These types of devices are much more expensive than standalone GNSS, they require more expertise to use them and require more preparation time to have
them up and running. However, these instruments allow to obtain accuracy at centimeter level. Thus, point positioning errors can be ignored. Such type of
instrument has to be used for measurement of Reference parcels to be then reported in the LPIS.
D. When to perform parcel area measurement?
It is reminded that where the information on the parcel's boundaries in LPIS
and/or in the GSAA reflects the field reality, no area measurement is needed. The
declared area will be considered as determined.
E. Determination of the parcel area, use of the technical tolerance If a measurement is done, a tolerance can be applied in most cases to take into
account the uncertainty of the tool used. If such, then where the absolute
(unsigned) difference between the measured and declared area is greater than
the technical tolerance (expressed as an area in hectares to two decimal places),
the actual area measured through physical measurement will be considered
determined.
In the alternative case i.e. when the declared area is within technical tolerance of
the measured area (below reported as the confidence interval) the area declared
will be considered as determined.
As from 2015, a unique buffer tolerance of maximum 1.25 m shall be used for
area measurement related to agricultural parcels. This area tolerance is calculated
by multiplying the parcel outer perimeter by the unique buffer tolerance value.
F. Minimum size The minimum parcel size applicable in a Member State has to be checked. Parcels
found below this minimum parcel size are not eligible for aid and should be flagged
with an appropriate code.
G. Parcel area measurement in the frame of “greening” Information provided so far remain valid for parcel area measurement to be
performed in the frame of crop diversification and also for EFA crop like features
(i.e. nitrogen fixing crops, catch crops).
These rules do not apply for all other area measurements to be done on
the frame of EFA (landscape features). For such areas please refer to the
technical specification provided in the Technical Guidance document on the On-
The-Spot Check of Ecological Focus Areas requirements (DS-CDP-2015-09-
FINAL).
29
H. Afforested areas - area measurements Several of the rural development support measures are based on area, and a few
of these concern afforested areas. These may be the case for first afforestation of
non-agricultural land, Natura 2000 payments, and forest-environment payments.
Since measuring in afforested areas is in general difficult (with GNSS due to the
risk of signal loss, on ortho imagery because parcel boundaries may not be
visible), MS are encouraged to look for individual solutions adapted to their
landscape, LPIS reference system and available data (number of claims, existence
of professional survey tools). Validation tests of GNSS equipment (with adapted
settings) may also be carried out. In such situation, MS may define appropriate
tolerances, which shall in no case be greater than twice the tolerances set for
parcel area measurements for OTS checks in first pillar. This means that MS may
use a buffer tolerance of maximum 2.5 m applied to the perimeter of the parcel,
with an absolute maximum of 2.0 ha.
I. Diagnosis and codes for area measurement check At the end of the OTS check process (i.e. after the pre-CAPI check, the CAPI or
RFV), each claimed parcel should be assigned at least one code concerning the
area and a retained area.
The roles of the technical codes are the following:
• Trace the work of the operator (e.g. for quality control purposes);
• Allow to determine the retained area for each claimed parcel;
• Describe the area error found;
• Allow a posteriori analysis and identification of particular problems (e.g.
high occurrence of a given code in a region).
When the area measurement is done, the comparison of is done with the declared
area taking into account the defined single buffer tolerance. In cases where the
agricultural parcel is composed of several cadastral parcels, computing the
tolerance at the level of internal cadastral parcel would lead to the application of
an excessive technical tolerance. Therefore, the single buffer tolerance has to be
applied on the perimeter of the agriculture parcel only or on the external perimeter
of contiguous agricultural parcels.
Area measurements always have to be done to limit the artificial elongation of
parcel perimeter. See following illustration.
30
A series of “standard” codes have been defined in relation to specific conditions.
Codes have been defined together with diagnosis of Land use / cover aspects. For
the details on codes, please refer to the dedicated chapter in Land use / Land
cover checks.
J. Capping to the reference area As a general rule the area(s) retained for the Basic payment Scheme (BPS, SAPS)
should not exceed the maximum eligible area of the corresponding LPIS reference
parcel.
A similar cap applies in the case of a reference parcel containing several
agricultural parcels. In particular, when these parcels are declared by different
farmers, a proportional reduction of the retained areas may be applied. In any
case, the contractor is referred to the instructions of the national Addendum.
However, if the LPIS reference parcel contains several crops eligible for different
area-related aid schemes, capping to the LPIS area applies individually for each
scheme.
6.2 Land use / Land cover check
The purpose of on-the-spot checks linked to land use / land cover will consist in
checking at least:
- the declared land use to the extent requested by the regulation (BPS,
SAPS, Greening, VCS, Rural Development support …);
- the number and/or position of trees and other features where necessary;
- the respect of the cross compliance requirements and particularly of the
Good Agricultural and Environmental Conditions (GAECs).
Classical field visit
Field inspection should be undertaken by operators having a very good knowledge
or specifically trained in eligibility and cross compliance rules, and crop
identification. Indeed, the check, for instance, of compliance of mixture of crop
species in catch crop parcels may require very good skill in plant identification and
taxonomy.
31
Field visits should be made at the best possible timing for identifying the crop(s),
assessing its extent and/or assessing the quality management of the parcel.
Field visit documents such as maps for the overall location of the parcels and
detailed location documents (e.g. parcel boundaries overlaid on a VHR image)
should be in possession of the controller. Alternatively, navigation systems based
on GPS and systems allowing the display of images, vectors and data on a mobile
computer in the field may be used.
It is recommended to take digital photographs of the parcels visited and especially
for parcels with problems, and stored in a database. Each photo should come with
its geo-location, a parcel identification number, date and time of the image
shooting, identifier of the operator. These pictures can be presented to the
applicant or an auditor in a follow-up meeting, thus reducing the number of follow
up field inspections to a minimum.
Finally, predefined codes should be used to report on the actual land use/land
cover and any anomaly found.
A. Use of ortho-rectified imagery (CwRS) Land use / land cover aspects may be checked by Computer Aided Photo
Interpretation (CAPI) of current year imagery often supported by the use of
automatic classification (supervised or unsupervised).
The CAPI of agricultural parcels will normally be carried out using at least one very
high resolution (VHR) image (satellite ortho-image or aerial ortho-photo with a
pixel size <0.75m (in some cases, like for the check of EFA landscape features
pixel size <=0.50m)) of the current year. In addition to the VHR image,
multitemporal high resolution (HR, or HHR) images may be provided upon detailed
justification by the administration. It is also recommended to use the Copernicus
Sentinel 1/2 sensors whenever suitable.
Depending on the agricultural land characteristics, staff availability, importance of
the different schemes, some Member States may opt for a CwRS method based
on the use of one VHR imagery to perform area measurements checks. Then, all
checks concerning Land Use / Land Cover aspects will be done through Systematic
Rapid Field Visits.
In the former SPS/SAPS CAP rules, the need to identify individual crop types was
not very strong. With the entry into force of the ‘greening’ and especially the crop
diversification requirements, there is now a strong need to identify crops at parcel
level (as it was the case during the ‘couple payments’ CAP period i.e. 1992-2004).
Methodology
The interpreter’s work could be summarized as follows:
· detect non eligible features (water, building, forest) paying attention to
features put in the ‘negative list’ in the considered Member States;
· pay special attention to trees on arable land/ permanent grassland (100
trees/ha eligibility criteria);
· check the crops subjected to coupled payments;
32
· Where the Administration has decided to use remote sensing for the control
of the GAECs, all declared parcels of the holding should be inspected;
· Where the Administration has decided to use remote sensing for the control
of 2nd pillar (land cover related Agri Environmental Measures), all parcels under
contract should be inspected.
We remind that interpreters should be specifically trained for checking all schemes
related elements to be checked though imagery. They should be proven with
common OTS checks guidelines.
During the photo-interpretation stage the interpreter must be able to
simultaneously display several images (e.g. a VHR ortho-imagery and a
multitemporal suite of HR ortho images) and also the vector and alphanumeric
data for each application. They should have the possibility to display on-screen
examples of parcels for which a preliminary ground survey has been done.
In case image data with more than 3 bands are used, it is advisable to select the
band combinations that contain the most significant information. This usually
includes the near-infrared band, the mid (or short-wave) infrared and one of the
visible bands, although the classical false colour composite (near-infrared, red,
green) is in general sufficient for checking the crops/uses that need to be
discriminated. The use of multi-temporal index images (e.g. NDVI image) is
another option.
In case the crop check is supported by an automatic classification, it is
recommended never using the result of classification ‘blindly’ and always have a
step of visual inspection of all parcels to confirm the diagnosis. When
classifications are used, the classification process should be documented.
Explanation should be given on how the classification results are used in the parcel
categorisation (e.g. as an ancillary image layer helping the interpreter or as
automatic parcel flagging).
BPS/SAPS land use / land cover check
In practice, land use check for BPS/SAPS will consist in checking compliance of
the parcel with eligibility conditions. Indeed, it could be said that the CAPI work
will mainly consist in focusing on the identification of non-eligible land (built up
zones, transport network, ‘negative list’). Interpreters should be trained in
detecting these elements. It may also be interesting to develop classification
algorithms for the detection of these crops or land covers. Although parcels not
claimed will have no impact on the diagnosis, checking these parcels allows to
better check claimed parcels (in case parcels not claimed share a reference parcel
with claimed parcels) and to train the interpreter on specific crops (e.g. crops that
may not be eligible for BPS/SAPS).
Control of Cross Compliance
Remote sensing data may be used in two ways for the control of cross compliance:
- Use of RS for a partial control of the GAEC.
- Use of RS as a support for the selection of the cross compliance sample
(risk analysis).
33
CwRS as a partial control of cross compliance: This approach may be envisaged
for the GAEC (or SMR) that may be checked on satellite or airborne imagery. This
is the case for instance for the maintenance of a soil cover during winter, the
prohibition of burning cereals stubble, the maintenance of some landscape
features etc… According to the minimum requirement defined for a given act or
standard, MS may decide to use RS images to check specific conditions (e.g.
requirements that need to be checked during autumn or winter).
In practice, during the photo-interpretation of the satellite imagery the CAPI
operator will flag any case of possible non-compliance (e.g. doubtful land use)
with an appropriate code.
Also cases of non-compliance in respect of some GAEC that would be observed
during a RFV should be reported to the Administration.
CwRS as a support for selecting the cross compliance sample: On the CwRS OTSC
sample or on the whole area covered by the HR image, an automatic classification
(refined by CAPI) could provide a list of parcels potentially in breach with some
GAEC that can be checked with RS. The corresponding dossiers may hence be part
of the risk based sample for the controls of cross compliance of a given control
body (the “1% sample” per competent authority).
Administrations should clearly describe the option(s) retained (no control of cross
compliance with RS, use of RS for partial control or for risk analysis) for each of
the control zones. If relevant, the GAECs to be checked and the criteria to be
assessed should also be described as well as the specific imagery / processing
requested (e.g. SAR imagery in winter for the detection of bare soil).
Voluntary Coupled Scheme check
Methodologies to identify crops under VCS scheme are the same as the ones
needed in the frame if crop identification to check compliance with the
‘diversification’ requirements. So for this section please refer to the ‘crop
diversification OTSC technical guidance’ (DS-CDP-2015-08–FINAL).
Check of diversification
For this section please refer to the dedicated ‘crop diversification OTSC technical
guidance’ (DS-CDP-2015-08–FINAL).
Check of Ecological Focus Areas
For this section please refer to the dedicated ‘Technical guidance for the On-The-
Spot check of Ecological Focus Areas (EFA) requirements’ (DS-CDP-2015-09–
FINAL).
B. Codes for diagnosis In classical Control with Remote Sensing (CwRS), i.e. the control performed in the
office mainly, diagnostic rules are applied at parcel, group and dossier levels. The
objective of these rules is to separate the dossiers that will need a field visit from
those which are considered as correct and therefore do not need any follow-up
action (for the points that could be checked by remote sensing).
34
Several codes may be used simultaneously if necessary. When several codes are
assigned to a parcel, the retained area and land use should correspond to the least
favourable condition.
Some codes are likely to change after a rapid field visit (if this option is chosen).
In the latter case, it will be preferable to keep track of the two successive
situations: i.e. to keep the code(s) before and code(s) after the rapid field visit.
In the frame of the control of Cross-Compliance, specific codes should be applied
to flag parcels for which a breach to a specific GAEC or cross compliance issue is
observed or suspected during the CAPI process.
In the frame of control of greening conditions, specific codes should also be
applied.
A series of ‘standard’ codes have been defined and classified in three categories
(see table below):
- The Tx codes are assigned during CAPI to parcels not checked for some
technical reason independent from the interpreter (e.g. parcel outside the
image). As assigning a T code implies giving the benefit of doubt to the
applicant, these codes should not be assigned to parcels deemed doubtful
during CAPI.
- The Ax codes correspond to anomalies, in particular those related to
eligibility, and lead to the rejection of part or a totality of the parcel.
- The Cx codes are assigned to the interpreted parcels (i.e. checked parcels)
but for which the declared area (or crop) is not accepted by the interpreter.
Different rules apply for computing the retained area.
If relevant, several codes could be assigned to the same parcel. If both the
declared area and the declared crop group are accepted, the controlled parcel will
be coded as “OK”.
Additional codes may be defined by the National Administration to record specific
cases not described by existing codes (e.g. LPIS boundary to be updated, or codes
for other schemes). In order to avoid confusion it is preferable:
not to reuse already existing codes (by changing their definition);
to create new codes by subdividing existing codes: for example A31
(unknown cadastral reference), A32 (valid cadastral reference, but no
vector).
Moreover, the new code(s) should be connected to an existing category (T, A, C)
as much as possible.
Calculation of retained area: The last column of table below indicates which area
should be retained at parcel level and therefore transferred to the crop group
level.
35
Table of Standard codes related to the conditions encountered at the parcel level,
and proposed rules for the calculation of retained area
Observations at the parcel level Code Areas transferred to the
crop group
Parcel outside all current year images
Parcel outside control zone (i.e. VHR
zone)
Parcel covered by clouds, haze, snow,
or flooding (G4CAP ‘meteo’ flag)
T2
T3
T4
Use the declared area and
land use
Parcel declared or found as less than
the minimum parcel size set by the
Administration
A1
Give zero value to the area
Parcel (or part) claimed more than once
A2
Give zero value to (the
disputed part of) the area
Parcel or reference not found in the
LPIS
Area ineligible
A3
A4
Give zero value to area
Parcel ineligible for SPS or SAPS
C1
Give zero value to the
eligible area, except for
"obvious errors". If possible,
indicate the land use found
Parcel declared in one coupled crop
group, but found in another (area
correct)
C2 Use declared area
and observed land use
Land use correct, area outside
tolerance (over-declaration i.e.
declared > measured)
C3+
Use measured area
And observed land use Land use correct, area outside
tolerance (under-declaration)
C3-
Land use interpretation impossible or
parcel limit problem not resolved on the
image
C4
Give zero value to the area
Obvious error not covered by another
code
E1 Use measured area
And observed land use
36
Land use correct, area within tolerance OK Use declared area and
declared land use
C. Rapid field visits Rapid Field Visits (RFV) are intended as a means to check the land use / land cover
and possibly some cross compliance issues (GAECs) in the field without contacting
the farmer. They are and directed to problem or doubtful parcels identified during
the CAPI.
It is advised to take digital photographs of the parcels visited and stored in a
database with their location, so as to be presented to the applicant in a follow-up
meeting, thus reducing the number of follow up field inspections to a minimum.
Results of the RFV should be used for completion of the diagnosis at parcel level.
Field visit documents such as maps for the overall location of the parcels and
detailed location documents (e.g. parcel boundaries overlaid on a VHR image) will
have to be provided to the staff in charge of RFV. Alternatively, navigation systems
based on GPS and systems allowing the display of images, vectors and data on a
mobile computer in the field may be used. Predefined codes should be used to
report on the actual land use and any anomaly found.
In classical CwRS, RFV may be used to assess the quality of the diagnosis derived
from the imagery. In this case the diagnosis established before and after RFV
should be recorded.
D. Categorisation at scheme group level For each scheme group, the total declared area of the crop group (Dg) will be
compared to the total retained area of the crop group (Mg). In practice, the areas
declared and retained for all parcels claimed in a given scheme group are summed,
therefore allowing compensation between over-claimed and under-claimed parcels
of the same scheme group (if this compensation is allowed in the Member State
concerned).
Sorting of scheme groups into Accepts and Rejects
For a given scheme group, the following three cases may be encountered:
- A1: The declared area is equal to the measured area (Dg - Mg = 0).
- A2: The declared area is less than the measured area (Dg - Mg < 0). In
this case, the Administration will accept and pay only the claimed group
area.
- R: The declared area is greater than the measured area (Dg - Mg > 0).
The first two categories are considered as accepted. All scheme groups with a
declared area greater than the retained area (third category) shall be rejected.
Sorting of rejected scheme groups into minor and major rejects
As any rejected scheme group should be subjected to a follow-up action because
it incurs reductions or sanctions, a second test may be performed in order to sort
37
minor and major rejects. This test, which consists in comparing the discrepancy
(Dg - Mg) with some threshold to be fixed by the Member State, is useful when
the follow-up action varies according to the discrepancy. Anyhow, it can be
suggested to use as threshold (Dg - Mg) > 3% Mg or 2 ha. If yes, the scheme
group is classified as RMa (major) if not it is classified as RMi.
If the follow-up action is the same for all rejects (e.g. letter sent to the farmer
and field inspection in case of no reply within a number of days), sorting the
rejected scheme groups appears as unnecessary.
The follow up actions for minor and major rejects are of the responsibility of the
Administration. Attention has to be paid to establish a diagnosis for the BPS crop
group, indicative diagnosis may have to be recomputed accounting for the
payment entitlements before starting any follow-up action.
E. Categorisation at dossier level There are three steps in the categorisation of the dossiers: A conformity test; a
completeness test; and a final diagnosis per dossier combining the two previous
ones.
Conformity test
A dossier is accepted if all crop groups are accepted (i.e. Dg-Mg <= 0 for any
scheme group). The table below summarizes this test for Member States making
a distinction between minor rejects (all crop groups are minor rejects) and major
rejects (i.e. at least one crop group is a major reject) at dossier level. The
proposed coding (DMi and DMa) remains valid whatever the test applied for sorting
the rejected dossiers (e.g. fixed threshold in ha or monetary unit). If no sorting is
applied (i.e. all rejected dossiers are processed in the same way), the categories
DMi and DMa could be amalgamated into one category coded DR1.
For dossiers concerned by BPS scheme, the categorization (as Accept or Reject
and for the Rejects, as minor or major Rejects) will be considered as provisory as
long as the payment entitlements will not have been taken into account. This
provisory categorization may be used as an indicator of the quality of the
application.
Input Test Dossier conformity test
Dossier codes
Yes No
The
whole
dossier
D1
All crop groups accepted (Dg-Mg <= 0) DA1 -
If at least 1 scheme group is rejected, dossier
is rejected DMi DMa
38
Are all rejected crop groups coded RMi ?
All
rejected
groups are
RMi
at least
1
rejected
group is
RMa
Completeness test
The purpose of the completeness test is to avoid accepting’ a dossier which has
been checked on a too restricted extent due to technical problems, i.e. T codes.
In such a case, the dossier is considered as not having been controlled by remote
sensing. If the dossier was part of the control sample, it has to be completed it in
the field, i.e. field inspected.
A dossier will be categorized as "complete" if the percentage of parcels with T
codes with respect to the claimed parcels is lower than 50% (Cf. Table below).
Input Test Dossier completeness test
Dossier codes
Pass
(complete)
Fail
(incomplete)
Area
retained
for: the
whole
dossier
D2
(# of parcels with T codes) / (# of claimed
parcels) <= 50% and (total retained area of
DC DI
T coded parcels) / (total retained area) <
20% and at least one parcel per crop group
In order to improve the efficiency of the control, applications sharing a reference
parcel with any application from the control sample may be included. This
recommendation is valid for any type of OTS check (physical inspection or CwRS),
and particularly for checking joint cultivations, but is probably easier to apply in
CwRS than in physical inspection. Such ‘ancillary’ applications are likely to be
incomplete and should hence not be completed in the field, in contrast with the
applications from the control sample. However, although very partially checked,
these applications could be rejected on the basis of irregularities found on the
parcels checked.
F. Final diagnosis at the dossier level The final diagnosis summarizes the diagnoses of the conformity and completeness
tests at dossier level. This table below proposes a general diagnostic code per
dossier and describes a possible follow-up action to be undertaken for rejected
crop groups or incomplete dossiers. It is reminded that incomplete dossiers that
were part of the initial control sample have to be completed in the field. In some
39
Member States, the contractor may be in charge of the RFV necessary to complete
the dossier (Cf. National Addendum).
The general diagnostic code proposed takes account of the distinction between
dossiers rejected for minor and major discrepancies. If such a sorting is not used,
the diagnostic codes can be simplified (e.g. DR7 and DR8 for rejected complete
and rejected incomplete respectively).
Whatever the diagnosis at dossier level, Member States may decide to manage
parcels outside tolerances by appropriate administrative procedures, in particular
if the anomaly originates from the LPIS.
A dossier categorized as incomplete will be counted and paid to the contractor if
it has been processed and photo-interpreted normally. It neither will be counted
nor paid if it appeared incomplete before the digitization and the photo-
interpretation.
Test Conformity Completeness Code Conclusion
D5 Pass Pass
(complete) DA5 Dossier accepted by remote sensing
D6 Pass Fail
(incomplete) DI6
Dossier not controlled with Remote
Sensing; the parcels which have caused
the dossier to be incomplete are verified
in the field
D7
Fail due to
small
discrepancy
only (DMi)
Pass
(complete) DR7p
Dossier ‘rejected’; all the rejected crop
groups being RMi, an appropriate
administrative procedure may be used to
notify the farmer of the correction
D8
Fail due to
small
discrepancy
only (DMi)
Fail
(incomplete) DR8p
Dossier ‘rejected’; the parcels that caused
the dossier to be incomplete are verified
in the field; the opportunity can be taken
to check rejected scheme groups (in case
no appropriate administrative procedure
has been applied)
D7
Fail due to
large
discrepancy
(DMa)
Pass
(complete) DR7f
Dossier ‘rejected’; an appropriate
administrative procedure may be used to
notify the farmer of the correction, but
usually the rejected crop groups are
verified in the field
D8
Fail due to
large
discrepancy
(DMa)
Fail
(incomplete) DR8f
Dossier ‘rejected’; both the rejected
scheme groups and the parcels that
caused the dossier to be incomplete are
verified in the field
40
6.3 Production of OTSC report
Although there is no OTSC report template currently provided by the Commission,
Article 41 of Regulation (EU) No 809/2014 lays down the mandatory elements of
the OTSC report. Accordingly, MS are recommended to develop and use a common
digital control report. This digital form should contain a predefined list of required
information.
In addition to the elements foreseen in the above-cited Article, it is advisable that
the report contain at least:
- Names of person(s) conducting the inspection;
- Date of inspection;
- Measurement tool used along with its specific settings (e.g. GNSS
measurement in vertex or continuous mode, scale of CAPI digitalization);
- Detailed summary of inspection findings (parcels okay, parcels flagged or
rejected, documentation and codes for the rejection, screen shot and/or
field photograph of the problem found, geo location of the parcels, possible
follow-up action)
- - Document all parcels for which an LPIS concern has been identified and
for which a follow should up be done by the competent LPIS custodian;
- - Document all parcels for which an EFA concern has been identified and for
which a follow should be done by the competent EFA layer custodian;
- …
For all information that is considered as mandatory (e.g. inspector name) it is
advised to prevent the form from until these fields are filled.
• Sign and certify each inspection report.
Each inspection report must be signed and certified by the permittee to be
considered complete. Where your inspections are carried out by a contractor or
subcontractor, it is recommended that you also have the form signed and certified
by the inspector, in addition to the signature and certification required of the
permitted operator. The template report should include a signature block for both
the administration and the farmer.
• Retain copies of all inspection reports with your records. These reports must
be retained for at least 10 years.
7 Analysis of the campaign – Act for the next campaign
In this section, some statistical procedures and other good practices are suggested
in order to take the best advantage of the control results and thus to prepare the
next campaign year in the optimal conditions.
7.1 Quality Controls
It is important to implement a quality management in all OTSC procedures. This
quality control process be can performed internally and/or targeted to the
contractor. Any contractor is required to carry out an internal quality assurance
41
which will result in quality control records. These records should be maintained for
inspection by the administration.
It is also advised to perform an independent external quality control. This was
formerly carried out by the Commission on one control zone per contractor. As
from 2009, the JRC stopped this activity.
With the entry into force of the new CAP legislation in 2015 and the resulting
impacts brought on OTSC processes, the JRC proposes to reintroduce a QC
process.
To do so, it is asked to any MS Administration to volunteer to provide a set of data
(i.e. LPIS and interpreted vectors, orthorectified imagery, orthorectification
Quality Control Records (QCRs), declaration data, measured and retained areas
at parcel level) corresponding to one CwRS zone.
Ideally, the quality controls should be spread on the whole controlled zones. This
would also enable us to check the homogeneity of the controls over the MS.
However, for both technical and practical sake, it is accepted to limit the QC to
only one control zone per contractor.
As a general rule, it is recommended for quality control reasons to verify in the
field a minimum number of the accepted dossiers from the selected control zone.
As dubious dossiers are automatically checked in field, it is considered that there
is no need to check them again during the QC.
For each control zone, this number is determined by the standard ISO 2859-
1:1999 (AQL set to 1,5% with a simple sampling plan and normal inspection).
Lot size Sample size Acceptance
number
Rejection
number
2-8 2 0 1
9-15 3 0 1
16-25 5 0 1
26-50 8 0 1
51-90 13 0 1
91-150 20 1 2
151-280 32 1 2
281-500 50 2 3
501-1200 80 3 4
1201-3200 125 5 6
3201-10000 200 7 8
10001-35000 315 10 11
42
A similar quality control should be carried out to assess the suitability of CwRS for
checking the GAECs, CD and EFAs requirements (and possibly SMRs related to
environment) that the MS have decided to check with RS. In particular, during the
inspection of the farms belonging to the GAEC sample, it is recommended to
perform rapid field visits on a sample of OK parcels in order to ensure that no
anomaly was overlooked by CwRS.
In addition, the Member States have the responsibility to carry out an external
quality control of the contractor's work.
At the end of this QC procedure, any inconsistence should be documented and the
causes of such discrepancy should be identified (e.g. lack of training, tiredness of
the inspector, workload …) in order to prevent future errors.
7.2 Study of residual errors
The error rates are computed as the ratio of area not found on area declared. As
those error rates are sample-dependent, they must be considered uncertain. Thus,
MS are recommended to analyse those results as because of their randomness,
they can reliably represent the reality.. It is considered not sufficient to verify
numerically that the estimated error rate is below some threshold (2% is typically
the accepted limit). A proper statistical test may be set out to address the question
“Is this error rate smaller than the 2% limit ?”. Similarly, when comparing two
error rates, their respective uncertainties may be taken into account in a test “Is
the error rate A smaller than the error rate B?”.
In this regard, the standard deviation(s) of the error rate(s) can also be estimated
based on the sample(s). The formula for this estimation generally depends on the
sample design(s). Specific formulas are detailed in Annex 3 as a recommended
means of analysis.
The exhaustive control of a zone (e.g. by CwRS) may be considered as a sample
of the whole population but is not a sample of this particular zone. Consequently,
the error rate computed on this particular zone is exactly known. No uncertainty
should be considered. In particular, the corresponding standard deviation is equal
to zero.
7.3 Analysis of campaign
Information gathered though the campaign and especially based on the number
of errors revealed or not revealed, taking into account the quality control phase
should help administrations to reflect upon the setting of the OTSC
methodology(ies) through:
- Considering the appropriateness of the various control methods applied in
respect of the particularities to be controlled
- Considering the links of the different elements within IACS (GSAA,
administrative controls, OTSC, sampling, number and type of errors, pre-
established information etc…)
- The appraisal of efficiency of the methods: imagery and/or field;
43
- Detailed study of photo-interpretation and automatic classification
processes; software and personnel, photo-interpretation keys;
- Analysis of the geometric and radiometric corrections of imagery;
- Analysis of the imagery use (usefulness of images ordered, relevance of
acquisition windows, % of RFV due to photo interpretation problems …);
- Analysis of measurements made with other tools (mainly GNSS devices);
- Analysis of the working timetable and “bottlenecks”;
If relevant, analysis of share of work between administration and contractor
partners; Estimate of the total number of dossiers processed each day;
- usefulness of ground data collection;
- …
‘Lessons learnt’ should also be combined with the outcomes of the LPIS Quality
Assessment.
Then it is advised to produce an assessment report or a textual analysis out of
these findings;
Then, from the analysis above, it is also advised to produce a remedial action plan
or textual description of remedial actions to be implemented for the following
OTSC campaign.
44
8 Annex 1: Details of image radiometric correction
Overall clipping of histogram
This metric is closely related to the visually detectable loss of information over
dark or bright objects that is caused by suboptimal image capturing or subsequent
processing. Controlling that the available dynamic range is efficiently used with
no undesirable saturation helps considerably for the preservation of the original
information content needed for the correct manual photointerpretation (CAPI).
The calculation of this metric is straightforward, monitoring whether the rate of
pixels belonging to the first and last 5 bins of the luminosity histogram are each
less than 0.5% of the total (recommended). In addition, it might be worthwhile to
also apply this radiometric ‘clipping’ metric separately for each individual channel
(MS and PAN), in the case a VHR bundle ortho-product is used. This is especially
valid for the NIR band, since it is a key contributor for the false-color composite.
This NIR doesn’t contribute to the luminosity value, and is therefore not assessed
by default. Checking the NIR band ‘cut-off’ of the VHR satellite sensors is
important as we sometimes observe an over-exposure (saturation) of the grey
values for the NIR band over smooth surfaces with high reflectance (partly due to
specular effect). More often, such cutting off can also be a result of the rescaling
to 8 bit (see images below).
45
Histogram Peak
The aim of this metric is to ensure that the overall lightness of the image is
optimized. It controls whether the minimum and maximum extents of the
histogram peaks of the different spectral bands are within the recommended +/-
15% range from the median value (it is 128 for 8 bit range). Indeed the exact
placement and shape of the histogram can vary significantly depending of the
image content (important factors are lighting conditions, land cover type,
topography, etc.). Therefore, the recommended thresholds cannot be considered
as absolute requirements. The aim is not to force’ the histogram to fit on or
converge over a common spot. We could rather apply a shaping function or specific
LUT (look-up table) stretch to achieve more balanced visual appearance on screen.
Although dedicated to the MS part of the orthoimagery, this metric might be
relevant also for the PAN band as well.
Colour Balance
The purpose of this metric is somehow similar to the previous one. It provides a
simple quantitative method for assessing whether the tonal neutrality over the
46
orthoimage is met. It can be argued whether or not such numeric verification is
needed, since the qualitative visual check often provides good estimates. The
problem is that this visual check is made against reference data, which is often
not available, or (if present) is not always reliable. In addition, qualitative color
balance evaluation is performed only under proper lighting conditions on display
systems that are calibrated and have the correct tone scale work station conditions
that cannot be always fulfilled. The proposed metric checks whether the difference
between the minimum and maximum digital counts of the RGB pixel values of a
given triplet is less than the recommended 2% of the total available bit range (5
bins for an 8 bit image). Even if the metric is very simple to apply, there are
certain issues with respect to its representativeness:
The first issue is that the chosen triplet should belong to a spectrally neutral
object, such as paved roads or building roofs. The results highly depend on the
quality and representativeness of the sample selected. In principle, we expect a
neutral object to have the same reflectance intensity along the visible wavelength
range, resulting in rather achromatic (grey) color appearance. But a reliable visual
assessment of such neutrality could be jeopardized by the same negative factors
that have impact on the qualitative visual assessment (lighting conditions,
calibrated monitor, etc.).
The second issue is that the metric is based on measuring the colour
balance on, albeit well-representative, still isolated spots of the image. It would
make sense also to check the homogeneity of the object where the triplet is
located. The ITT report for NAIP specifies a different metric and threshold for the
difference between triplets across an image, within the same object type or cover
type (see page 23 of the report). In addition, same report suggests also another
metric controlling the colour saturation that is consistently applied on the whole
image (see point 4.3 on page 14). Further research is needed to reveal which
metric is more relevant in the context of LPIS and CwRS.
Finally, we need to be aware that for VHR satellite data the degree of colour
preservation the original MS image depends considerably on the rescaling and
pan-sharpening methods. Often the appropriate sharpness of the PAN is achieved
at the expense of poorer colour recovery or vice-versa.
On picture below, you can see a 16bit image (1), rescaled to 8-bit with Min-Max
method (2) and with StDev method (3). Differences for the RGB triplet are within
2% for (2), but not for (3).
47
The picture below shows the same image pan-sharpened with 3 different
algorithms and showing RGB values for the same triplet. We can see the DN values
inside the RGB triplet vary depending on the pan-sharpening method.
Noise
Noise is defined as non-image related variations in intensity and can have an effect
on the interpretation of an image. The SNR method proposed in the JRC
orthospecifications is based on the simple ratio between the mean DN value and
the Standard deviation of the DN values over a given uniform neighborhood (often
calculated on the base of a 3X3 moving window). It should target areas that are
prone to be noisier. The Rose criterion (after Albert Rose) states that an SNR of
at least 5 is needed to be able to distinguish image features at 100% certainty
(see the Rose Model). However, it is not yet clear whether this method of
calculating the SNR is the optimal in the CAP context. The JRC Orthoguidelines
propose that the Standard Deviation alone as a global statistic over the whole
image gives sufficient estimate on the amount of undesirable noise. The
48
recommended upper limit of the Standard Deviation is 12. Further analysis can be
made in selected homogeneous/inhomogeneous areas. Some contractors propose
as well the PSNR (Peak signal-to-noise ratio) as an alternative measure.
Furthermore, we need also to take into account that the acquisition systems of
VHR sensors and the new digital aerial cameras are designed and adjusted in a
way to ensure that in the majority of cases, a sufficient ratio between the
meaningful input signal and the background noise. In such case, the SNR metric
should be probably focused on assessing the quality of the different image
compression methods after the ortho production.
Contrast
Contrast is one of the basic and most essential metrics monitoring the amount of
information that an image can provide. High contrast images allow better
distinction of land cover feature and facilitate at great extent the work of the CAPI
operator. The contrast in the JRC orthospecifications is expressed through the
coefficient of variation, which is the Standard Deviation of the DN values as a
percentage of the available grey levels. It should preferably be in the range of 10
% - 20%. It should be calculated separately for each band (MS and PAN), in case
if VHR bundle ortho-product. The ITT report suggests also an alternative contrast
metric (calculated on the luminosity) given in point 4.2 of the report. It derives
the number of bins between the minimum and the maximum of the cumulative
luminosity histogram function. In principle the modern digital sensors (aerial or
satellite) with high radiometric resolution (11bit or more), do not experience
problems providing images with sufficient contrast, since the adjustment of their
dynamic range during the acquisition and pre-processing ensures that the full
histogram extent is obtained for each channel, leaving some margins at the tail
for further post-processing. However, the dynamic range of the input imagery
depends not only on the light source intensity and target reflectivity, and can
become an issue for imagery with noticeable percentage of cloud cover or haze.
Luminance Histogram
Luminance histograms are more accurate than RGB histograms at describing the
perceived brightness distribution or "luminosity" within an image. Luminance
takes into account the fact that the human eye is more sensitive to green light
than red or blue light. The luminance histogram also matches the green histogram
more than any other colour.
In order to produce a luminance histogram, each pixel is converted so that it
represents a luminosity based on a weighted average of the three colours at that
pixel. This weighting assumes that green represents 59% of the perceived
luminosity, while the red and blue channels account for just 30% and 11%,
respectively. Once all pixels have been converted into luminosity, a luminance
histogram is produced by counting how many pixels are at each luminance,
identically to how a histogram is produced for a single colour.
An important difference to take away from the above calculation is that while
luminance histograms keep track of the location of each colour pixel, RGB
histograms discard this information. A RGB histogram produces three independent
49
histograms and then adds them together, irrespective of whether or not each
colour came from the same pixel.
When assessing the luminance of VHR satellite data in particular, one should pay
attention to the fact that the luminosity histograms generated for the original
multispectral image and the resulting pan-sharpened product are different. This is
mainly due to the "non-linear" contribution of the panchromatic component in the
pan-sharpened product (see the example below)
50
9 Annex 2: JRC "Area measurement tool validation method"
The validation method is designed to determine the inherent tool error
(accuracy). It should be set to limit as much as possible other possible errors
(e.g. bad use of instrument, non-respect of parcel border …). It is not a
proficiency test. The result of the validation is strictly related to the tested
method of measurement and not only to the device. Therefore the certificate or
the validation report remains valid as long as the operators use the tested
method.
The quality of a measurement tool can be characterized by a number of
parameters such as its bias, precision and accuracy. Assuming there is no bias,
it can also be characterized by its reproducibility limit, which is the parameter
used to determine the technical tolerance.
The buffer validation method for both GNSS devices and ortho-imagery, is
summarised in the following flow chart:
51
A. Data collection
A. 1. For validation of GNSS devices
The test shall cover hardware, software, settings and method
The test parcels should have unambiguous borders to ensure that all
measurements cover the same object (for instance the borders could be marked
with wooden sticks with a density of at least 1 peg per 25m); objects should be
of variable sizes (at least covering the range over which the GNSS should be
working, for instance between 0.2ha and 4ha) and shapes (at least one
elongated parcel should be included).
Number of fields
The more fields chosen for the test, the more reliable the assessment: more
data collected give more points on the receiver characteristic curve. It is
recommended to take at least 6 fields with sizes spread along the typical size
range of the country.
Shape
The shape of the fields should vary from very simple shapes (e.g. rectangular)
to some irregular shapes with high perimeter to area ratio
Obstructions of the horizon
Validation test results conducted showed, a high impact of horizon obstruction
on the buffer, e.g. parcels with trees to the south often show a significantly
higher R-limit than parcels without tree obstruction. Selecting most test parcels
in an open horizon environment is likely to result in a (low) buffer which may not
be appropriate to the usual conditions in which the device is used.
Terrain characteristic and the type of cultivation is an important issue due to
possible disturbances of the satellite signal. In mountainous areas or on fields
(partly) bordered by trees, the “visibility” of the satellites may be limited, which
may result in higher measurement errors. If parcels with partially obstructed
borders are common in the region or country, such parcels should be included in
the test so as to reflect the average conditions of measurement in the
region/country.
52
Example of reference shapes set
Borders of the test field
The test fields should have easily accessible borders (no stones, bushes to cross,
marshy spots etc.) to allow operators to feel comfortable while moving around
the borders. This will reduce the impact of the operator on the result of the
measurements.
Reference value
Standard deviation of measurement repetitions should be estimated against a
reference area of the considered parcel. The reference area of the test parcels
should be established with a surveying tools or GPS RTK measurements. It can
also be taken from a LPIS reference area if a reference shape corresponds to it.
Repetitions
The GNSS constellation should be considered as relatively stable while collecting
data on each field. In other words, the time needed to measure one field four
times is short enough (normally 10 to 30 minutes, depending on the perimeter
of the field) to consider the satellite constellation as stable. The four
measurements taken in a short period of time, by the same operator, will allow
to derive the repeatability variance of the area measurements.
Runs (sets) of measurements
The revisit time for the GPS satellite constellation is equal to ~12h. In order to
make measurements under different constellations (i.e. different number of
satellites and different satellites in view), the different runs should start at
different times of the day. At least 1.5 - 2 hours should be left between two
successive runs so as to consider that the satellite constellations have changed.
53
The variance between runs of measurements will be used to derive the
reproducibility variance of the area measurements.
An example of organization of field measurements for 6 parcels with 2 operators
Settings
Perform the test with exactly the same settings that will be used during OTS
check work (max DOP, S/N ratio, logging interval).
Method
Perform the test with exactly the same method that will be used during the on-
the-spot- check (continuous logging of the points along the field border, or
log of the vertices of the field). In case of area measurement done by logging
of the vertices of the reference fields, the distance between two successive
vertices should not be greater than 25m. This is to “simulate” the natural
landscape measurement conditions, where the borders are rarely straight and
data are logged more frequently than when measuring rectangles.
However, in case of very elongated straight parcels (i.e. more than 400, 500
meters long) the distance between two successive vertices can be extended to
100, 150 meters.
As during the real field controls there are situations where the GNSS device is
used in continuous AND vertex measurement method validation for the both
methods is necessary.
Avoiding systematic errors
Operators should not disturb each other while measuring so if possible one
operator should measure one parcel at one time. If this is not possible, special
attention should be paid to the location of the antenna while passing each other.
In order to avoid potential systematic errors related to left/right handed
operators, the direction of walking when measuring fields should be both:
54
clockwise and anticlockwise: e.g.: for all the runs: 1st repetition – always
clockwise, 2nd repetitions always anticlockwise, 3rd – clockwise, 4th –
anticlockwise.
A.2. For validation of orthoimagery
For orthoimagery, since repetitions of measurement are less time consuming
than field measurement, the minimum number of parcels and repetitions could
be increased as follow.
Selection of at least 30 parcels. In order to ease the work, it is advised to
select parcels corresponding to LPIS reference parcels (reference area already
available). Otherwise, it will be necessary to measure the reference area on field
using a surveying tools or GPS RTK measurements.
Area measurements of these parcels performed by at least 6 operators (for
proper statistical analysis)
Each operator performing at least 4 measurements (repetitions) of each
parcel (for proper statistical analysis)
For what concerns the parcels’ selection:
Parcels selected should be a representative sample of the control area
zone (strongly related to the parcel structure)
Parcel sizes should cover the range observed in the control area
o S: small
o M: medium
o L: large
Parcel shapes should vary
o SF1: compact
o SF2: elongated
o SF3: very elongated
Some parcels should be selected with easily identifiable borders as to
avoid interpretation problems and lead to some parcels rejections later on
during the analysis
Remarks concerning choice of the parcel size and shape ranges
Before parcel set definition (size and shape) statistical analysis of the parcel
structure on the area to be controlled should be performed. In the first step
parcel areas are sorted and 5% of outlying values are discarded (percentiles:
97.5% and 2,5%). In the next step the remaining range is divided by 3 equal
parts (small, medium and large size). Parcel area and perimeter allow for Shape
55
Factor calculation (SF=(perimeter/4)^2/area). The same procedure should be
performed for SF (compact, elongated and very elongated parcels).
Example of parcel set (30 parcels):
good border: 15 parcels
o S and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
o M and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
o L and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
"fuzzy" border: 15 parcels
o S and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
o M and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
o L and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
Remarks concerning choice of scale for digitisation
Considering the pixel size of imagery to validate (i.e. <= 0.5m), the scale for
digitization should be around 1/1.000. Nevertheless, the scale should not be
fixed to 1/1.000 and should be adapted on a parcel per parcel basis.
Example of parcel border
Some examples are provided hereafter in order to illustrate the concepts of
‘easy border’, ‘fuzzy border’ and ‘borders leading to interpretation problems’.
56
Examples of ‘easy’ and ‘fuzzy’ borders that should be part of the parcels sample
Examples of parcels with limits difficult to delineate (photo interpret) and that
should not be part of the selected parcels sample.
Precise, commons and detailed instructions (the same as used for creation of
reference parcels – LPIS if parcel based LPIS available) have to be given to the
photo interpreters.
For field parcels, pegs are provided to clearly identify parcel borders. For ortho-
images it is not possible to provide the equivalent of pegs otherwise there will be
a risk that photo interpreters digitize these vertices/lines not doing interpretation
of the image. Therefore, as guide for interpretation, a shape surrounding the
parcel to be measured, has to be created in advance for all parcel in the sample,
and shown on the screen during the measurement.
57
As for the GNSS validation protocol, the reference area and perimeter of the test
parcels should be established (from LPIS if available or performing field
measurement using surveying tools or GPS RTK measurements).
B. Statistical analysis
Collected data
The results of the area measurements performed in validation process should be
collected like in Table A of ISO 5725. In each cell there is measured parcel area
(in this case in square meters). Each 4 repetitions define one set, or used in ISO
5725 one laboratory, or sometimes called one run. So one operator delivers 3
sets (in GNSS validation, see diagram above and table below). Level in ISO 5725
58
means in our validation procedure - parcel (6 parcels: A, B, C, D, E and F - 6
levels).
Table A
Basic statistics
Next for each parcel mean area and standard deviation for each set is calculated
(Table B and C in ISO 5725-2).
59
Outliers detection
The statistical parameters concerning the buffer tolerance value should be
calculated only for the dataset free of outliers. Therefore the collected data need
to be tested for outliers see flowchart below. Cohran's test is described in
chapter 7.3.3 and Grubss' test in chapter 7.3.4 of ISO 5725-2.
Cochran’s test checks variation of standard deviation between classes.
Grubb’s test checks variation of means between classes, standard
deviation is calculated between classes.
Grubb’s test for single observation checks variation of observed value
in class (standard deviation is calculated within class).
60
61
During the outliers' detection, follow the flowchart above and perform the tests
according ISO 5725-2
After the outliers removing tables: A, B and C are modified. It is recommended
to put the reason of outliers discarding in table A. Single observations or all sets
can be removed.
Calculation of repeatability and reproducibility
After outliers detection the uncertainty of parcel area measurements is
estimated.
ISO 5725-1
Repeatability standard deviation: the standard deviation of test
results obtained under repeatability conditions.
Repeatability conditions: conditions where independent test results are
obtained with the same method on identical test items in the same
laboratory by the same operator using the same equipment within short
intervals of time.
Reproducibility standard deviation: the standard deviation of test
results obtained under reproducibility conditions.
Reproducibility conditions: conditions where test results are obtained
with the same method on identical test items in different laboratories with
different operators using different equipment.
62
Verification of bias and influence of pooling factors
According the interlaboratory tests performed in the past and during ongoing
projects the significant, repeatable and rigid influence of parcel border quality on
the value of reproducibility was observed. Any other factors should not influence
the validation results. However we observed in some cases the bias, influence of
the operator, walking direction, measurements day, parcel area and size etc. But
generally we expect no bias and no influence of any factors presented in the
table below. Therefore we recommend performing bias test (basing on the
formulas in IS0 5725-2 chapter 7.4.5, ISO 5275-4 chapter 4.7.2 or T-Student
test). Influence of the other factors is recommended to verifying using ANOVA
analysis.
63
C. Buffer tolerance estimation
Repeatability and reproducibility standard deviations are for each parcels in
square meters. Reproducibility limit in area values [m^2] depends on parcels so
for the standardization it should be divided by reference perimeter to obtain
reproducibility limit in buffer values [m].
According ISO 5725-6 chapter: 4.1.4: when examining two single test results
obtained under reproducibility conditions, the comparison shall be made with the
reproducibility limit (in our case: buffer limit): RL=2,8 (sRj in buffer).
D. Classification of the buffer width for a measurement tool
Reproducibility limit calculated in validation process allows classifying the area
measurement method to the one of the following classes:
(1) "1.0m" for RL inside (0.75m, 1.0m];
(2) "0.75m" for RL inside (0.50m, 0.75m];
(3) "0.50m" for RL below 0.50m.
Mean value of repeatability in buffer for our example: 0,61 m so the validated
method is classified into class (4): buffer limit taken to the control should be:
0,75 m
E. Report from validation
Report from validation should include following informations:
(1) Validated equipment: type of GNSS receiver with software (type and
version), serial numbers of each entity, metadata about orthoimagery
(type, resolution, uncertainty from quality control etc.);
(2) In the case of GNSS receivers:
o device settings (elevation mask, max DOP, etc.);
o details about the validated method: vertex + number of logs per
vertex / continuous + logging interval;
64
o use of differential correction and type of correction;
o measurements with or without external antenna.
(3) reference parcels areas and perimeters, details about the
measurements method applied for reference measurement;
(4) design of parcels set;
(5) table A with some explanation if needed (notice about the not normal
procedure, applied equipment if changed or shared between operators,
gross error etc.);
(6) basic statistic before outliers discarding: table B and C;
(7) results of outliers testing: table A with the removed single
observations and/or all sets;
(8) repeatability and reproducibility standard deviation in [m^2] and in
buffer values [m];
(9) results of bias analysis and ANOVA analysis;
(10) buffer limit and class of the validated method;
Test data.
An example of xls file containing the collected data from an area measurement
validation test can be found on WikiCAP
(http://marswiki.jrc.ec.europa.eu/wikicap/images/4/41/Validation_test_data.xls)
F. Documentation needed when the statistical analysis is to be made or
validated by JRC
1. In case a MS decides to entrust JRC with the statistical analysis after data
collection, the following information should be sent to JRC for final analysis:
Report from validation of the test carried out by MS (points: 1-5). Last
four points (6, 7, 8 and 9) will be prepared by JRC (outliers detection,
repeatability and reproducibility standard deviation calculation, buffer limit
determination and result of method classification).
Detailed description of validation procedure (protocol) should be
delivered.
Raw measurements data
o In the case of GNSS - a copy of the measurement protocol
indicating parcel id, date and time, set, repetition, operator,
direction of measurement, area measured, perimeter measured.
o In the case of orthoimagery validation all vector files should be
delivered.
65
A technical report and the data will document the whole validation process; they
will help JRC to evaluate and analyze the data as well as to draw conclusions on
the tolerance to be used with that device and measurement method. The final
statement on the performance of the system will be prepared by the JRC on the
basis of the test results.
2. In case a MS decides to perform the validation tests and the statistical
analysis by itself a technical report, data and the sheets with the statistical
analysis (templates to be asked to JRC) should be sent to the JRC for validation,
final assessment and publication of the results on the JRC web page.
66
10 Annex 3: Recommended means of analysis of residual
errors
A. Testing the significance of a random error rate on one control zone
For an individual error rate (i.e. the error rate of a single control zone or the error
rate of the whole sample), the statistical test is rather simple. It is actually similar
to the test that is performed for the LPIS Quality Element 1b within the LPIS QA
exercise.
First, the standard deviation SR of the error rate R must be estimated. This
estimation depends on the type of sample, i.e., random or from the risk analysis
and also on the groupe of beneficiaries that this sample represents (i.e. one
control zone or the whole population of the MS).
If the error rate 𝑅 is computed from a random sample, one must use the following
procedure:
- For each individual dossier in the sample, compute the difference 𝑞𝑖 = 𝐸𝑖 −
𝑅 ∗ 𝐶𝑖 where 𝐶𝑖is the claimed area and 𝐸𝑖is the area not found;
- Compute 𝑆𝑞 as the sample standard deviation of the vector of 𝑞𝑖;
- 𝑆𝑅 is then computed as
𝑆𝑅 = √𝑁 − 𝑛
𝑁 − 1
𝑁 ∗ 𝑆𝑞
𝑛 ∗ 𝑇𝑜𝑡𝐶
where 𝑛 is the number of dossiers in the sample, 𝑇𝑜𝑡𝐶 is the total claimed
area over the considered zone (i.e. either the control zone or the full MS
territory) and 𝑁 is the total number of dossiers over the considered zone
(again, either the control zone or the full MS territory).
If the error rate 𝑅 is computed from a risk-based sample, the procedure for
estimating the standard deviation of the error rate is more complex. It will be
proposed in details with clear examples in the WikiCAP.
Once, the standard deviation 𝑆𝑅 is estimated, one must evaluate the upper limit
of error (ULE) with
𝑈𝐿𝐸 = 𝑅 + 𝑡𝑛−1,0.95 ∗ 𝑆𝑅
Where 𝑡𝑛−1,0.95 is the quantile at 95% of the Student distribution with n-1 degree
of freedom.
Finally, the conclusion of the test “Is the error rate R smaller than the 2% limit ?”
depends on which of the following cases is verified:
Cases Conclusions
𝑅 < 𝑈𝐿𝐸 ≤ 2% The error rate is significantly smaller
than 2%.
𝑅 ≤ 2% < 𝑈𝐿𝐸 There is not enough evidence that the
error rate is smaller than 2%.
2% < 𝑅 < 𝑈𝐿𝐸 There is enough evidence that the error rate is larger than 2%.
67
B. Comparing random error rates from different control zones For the comparison of two error rates from independent samples (e.g. error rates
from two different control zones, risk versus random samples), the preparation is
the same. Only the test differs has it tries to address questions such as “Is the
random error rate A larger than the random error rate B?” or “Are both random
error rates equal?”.
The random error rates 𝑅𝐴 and 𝑅𝐵 and their respective standard deviations 𝑆𝐴 and
𝑆𝐵 must be evaluated individually as described previously and accordingly to the
type of samples.
The statistic of the test will be computed as
𝑇 = 𝑅𝐴 − 𝑅𝐵
√𝑆𝐴2 + 𝑆𝐵
2
Then, according to the test that is considered, the conclusion of the test depends
on which of the following cases is verified:
Tests Cases Conclusions
“Is 𝑅𝐴 larger than 𝑅𝐵?” 𝑇 ≤ 𝑡𝑚,0.95 There is not enough
evidence that 𝑅𝐴 is larger
than 𝑅𝐵. 𝑡𝑚,0.95 < 𝑇 𝑅𝐴 is significantly larger
than 𝑅𝐵.
“Are 𝑅𝐴 and 𝑅𝐵 equal?” 𝑎𝑏𝑠(𝑇) ≤ 𝑡𝑚,0.975 There is not enough
evidence that 𝑅𝐴 and 𝑅𝐵 are different.
𝑡𝑚,0.975 < 𝑎𝑏𝑠(𝑇) There is enough evidence
that 𝑅𝐴 and 𝑅𝐵 are
different.
where the value m is computed as
𝑚 = (𝑆𝐴
2 + 𝑆𝐵2)2
𝑆𝐴4
𝑛𝐴 − 1 +𝑆𝐵
4
𝑛𝐵 − 1
𝑡𝑚,0.95 and 𝑡𝑚,0.975 are respectively the quantile at 95% and 97.5% of the Student
distribution with 𝑚 degree of freedom and 𝑛𝐴 and 𝑛𝐵are the sizes of the samples
A and B respectively.
C. Comparing two error rates: paired case For the comparison of two error rates from dependent sample (e.g. in case of
reprocessed dossiers during a QC or for testing the equivalence of CwRS and
classical checks), the procedure is slightly different. Instead of working on the
error rates separately, one must keep the original structure of the sample, i.e. the
sample is in fact constituted as pairs of observed area-not-found (e.g. a CAPI
measurement and a field measurement) on the same dossier.
68
First, the standard deviation 𝑆𝑅𝐷𝑖𝑓 of the difference of the error rates 𝑅𝐷𝑖𝑓 = 𝑅1 −
𝑅2 must be estimated. This estimation depends on the type of sample, i.e.,
random or from the risk analysis.
If the error rate 𝑅𝐷𝑖𝑓 is computed from a random sample, one must use the
following procedure:
- For each individual dossier, compute the difference
𝑞𝑖 = (𝐸1,𝑖 − 𝐸2,𝑖) − (𝑅1 − 𝑅2) ∗ 𝐶𝑖
where 𝐶𝑖 is the claimed area and 𝐸1,𝑖 is the area not found using the first
method and 𝐸2,𝑖 is the area not found using the second method;
- Compute 𝑆𝑞 as the sample standard deviation of the vector of 𝑞𝑖;
- 𝑆𝑅𝐷𝑖𝑓 is then computed as
𝑆𝑅𝐷𝑖𝑓 = √𝑁 − 𝑛
𝑁 − 1
𝑁 ∗ 𝑆𝑞
𝑛 ∗ 𝑇𝑜𝑡𝐶
where 𝑛 is the number of dossiers in the sample, 𝑇𝑜𝑡𝐶 is the total claimed
area over the considered zone (i.e. either the control zone or the full MS
territory) and 𝑁 is the total number of dossiers over the considered zone
(again, either the control zone or the full MS territory).
If the error rate 𝑅𝐷𝑖𝑓 is computed from a risk-based sample, the procedure for
estimating the standard deviation of the error rate is more complex. It will be
proposed in details with clear examples in the WikiCAP.
Once, the standard deviation 𝑆𝑅𝐷𝑖𝑓 is estimated, one must evaluate the lower and
upper limit of error (LLE and ULE) with
𝐿𝐿𝐸 = 𝑅𝐷𝑖𝑓 − 𝑡𝑛−2,0.975𝑆𝑅𝐷𝑖𝑓
𝑈𝐿𝐸 = 𝑅𝐷𝑖𝑓 + 𝑡𝑛−2,0.975𝑆𝑅𝐷𝑖𝑓
where 𝑡𝑛−2,0.975 is the quantile at 97.5% of the Student distribution with n-2 degree
of freedom.
Note that there are n-2 degrees of freedom and not n-1 as in the test on one
isolated error rate in previous section. Also, it is a two-sided test so there are two
limits: 𝐿𝐿𝐸 and 𝑈𝐿𝐸.
Finally, the conclusion of the test “Are the error rates R1 and R2 equal ?” depends
on which of the following cases is verified:
Cases Conclusions 𝐿𝐿𝐸 ≤ 0% ≤ 𝑈𝐿𝐸 There is not enough evidence that the
error rate are different.
Else There is enough evidence that the
error rate are different.
Europe Direct is a service to help you find answers to your questions about the European Union
Free phone number (*): 00 800 6 7 8 9 10 11
(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed.
A great deal of additional information on the European Union is available on the Internet.
It can be accessed through the Europa server http://europa.eu
How to obtain EU publications
Our publications are available from EU Bookshop (http://bookshop.europa.eu),
where you can place an order with the sales agent of your choice.
The Publications Office has a worldwide network of sales agents.
You can obtain their contact details by sending a fax to (352) 29 29-42758.
XX-N
A-x
xxxx-E
N-N
doi:xx.xxxx/xxxx
ISBN xxx-xx-xx-xxxxx-x
XX-N
A-x
xxxx-E
N-N