executive test suite explained: belgium - flanders...

LPIS Workshop: Applications and Quality, 6-8th October, 2009, Tallinn 1 / 20

Executive Test Suite explained:

Belgium - Flanders experience

Pavel Milenov, Piotr Wojda, Sara De SantisEC – Joint Research CentreInstitute for Protection and Security of the CitizenMonitoring of Agriculture Resources

Pieter Roggemans


Outline

• Overview of ETSObjective and basic conceptKey quality elements and measuresWorkflow: Sampling, Inspection procedure, reportingPerformance issues and future challenges

•Belgium - Flanders experience with ETSFeasibility, technical constraintsTime and resources neededObservations and suggestions

•Status of the ETS pilots


Objective of ETS

• unambiguous localisation of all claimed agricultural parcels• quantification of all eligible area for crosschecks

To measure the performance of the LPIS on the ability to fulfil two explicit LPIS functions:

Through evaluation of the level of conformance of a number of pre-defined and agreed LPIS quality elements

Which values are derived though a certain set of data quality measures, specific for each LPIS quality element


Quality Elements and Data Quality Measures

conformance level

10203 - LPIS eligibility rates

LPIS Sample Level

indication of the statistical spread and nature of the quality problem

Distribution of RP allowing payment on ineligible land

conformance level

10202 – LPIS non-conformant RP

LPIS Sample Level

conformance level

10102_2 - RP Area purityReference Parcel Level

informative10105 – RP Ineligible land cover features

Reference Parcel Level

indication of the statistical size of the quality problem.

Proportion of RP allowing payment on ineligible land

1.2

conformance level

10201– LPIS eligible areaLPIS Sample Level

informative10104_2 – RP landscape elements area


informative10104 - RP landscape elements


Informative10103 – RP diversityReference Parcel Level

informative10102 – RP true eligible area


maximum quantity of land that can be claimed by farmers (SAPS) or eligible hectares upon which entitlements can be activated (SPS)

Total eligible area1.1

ExpectationMeasuresObservation and Reporting Level

DescriptionQuality topic10999.doc


conformance level

10101 - RP positional accuracy


Rate of boundary segments with inaccurately positioned nodes

Boundary accuracyN/A

conformance level

To be developedIrregularities identified during the OTSC can reflect shortcomings of the LPIS

Relation with error rates identified during OTSC field inspection

1.7

conformance level

N/A in ETS claim registerthe occurrence and abundance of transactions originating from LPIScustodian

conformance level

To be developedthe occurrence and abundance of transactions originating from OTSCinspector

conformance level

N/A in ETS claim registerthe occurrence and abundance of transactions originating from farmer/

Effectiveness of the update processes

1.6

All unclaimed area identified

N/A in ETS: claim register

Proportion of claimed farmed land to the reference area

Claimed area inside the reference parcel

1.5

conformance level

10205 - LPIS critical defects

LPIS Sample Level

informative10106 - RP critical defects


Occurrence of non-compliances that obstruct the use of the parcel

Critical defects1.4

conformance level

10204_2 - LPIS number of anomaly causes

LPIS Sample Level

abundance of parcel categories based on the cause of the contained problem

Categorization of the RP allowing payment on ineligible land

conformance level

10204 – LPIS cause of anomaly

LPIS Sample Level

occurrence of parcel categories based on the cause of the contained problem

Categorization of the RP allowing payment on ineligible land

1.3

Quality Elements and Data Quality Measures (2)


Data Flow and Processes in the ETS

Inspect certain number of randomly selected reference parcels on the most recent orthoimagery


Model Conformance TestConformance Statement (ICS)

Abstract Test Suite (ATS)

Data Conformance TestExecutable Test Suite

(ETS)

Additional Information for Testing

Conformance Test Report

Analysis of results

Application Schema or

Feature Catalogue of the implementation

under test

Model Conformance TestConformance Statement (ICS)

Abstract Test Suite (ATS)

Data Conformance TestExecutable Test Suite

(ETS)

Additional Information for Testing

Conformance Test Report

Analysis of results

Application Schema or

Feature Catalogue of the implementation

under test

•Quality measures and procedures based on ISO

•Most recent orthoimagery as a reference Satellite Imagery collected under the CwRS program

compliant with the JRC VHR specifications for the CwRS

•Inspection based on manual photointerpretation and delineation•Flexible mapping rules

•Identification and quantification of features, based on the difference in land cover

•No link with ATS yet

•No follow-up visits on the field planned

Basic properties of ETS


Structure and Content of the ETS• ISO compliant data quality measures, at parcel and sample level – Annex I

ISO/TS 19138 and ISO 19114

•Inspection procedure at the Reference Parcel – Annex IIUML Activity DiagramFlow of events and non-functional requirements

•Application of the land cover concept to describe eligibility of land – Annex III

The concept of eligibility profile introduced

•ISO based Sampling Scheme – Annex IVISO2859-2

•SQL statements for database analysis and reporting


ETS Sampling scheme•A population of reference parcels defined for each country/region

Isolated lot of reference parcels•A minimum number of reference parcels to be tested

Sample size based on ISO 2859/2-1985 (Procedure A, Limiting Quality = 2%)

•A larger number of reference parcels Sample pre-selection based on clipped reference parcels


Practical sampling procedure•Reference parcels are represented by points

located inside each parcel•Reference parcels are clipped with CwRS zones

only zones, where VHR data is successfully acquired

1 2503 75080 049764 122

Sample sizePre-selectionClippedLot


Data Quality Measures - RP true eligible area


ETS Inspection Procedure• Select the reference parcel from the selection list•Locate the reference parcel•Decide if it can be inspected or not. If yes, proceed with the inspection. If not, select the next one from the ordered list

• Identify and delineate the eligible land cover features on the land represented by the reference parcel. Derive the area.•Identify the eligible landscape features on the land represented by the reference parcel. Derive the area.•Identify the ineligible land cover features on the land represented by the reference parcel. •Identify if the reference parcel has a critical defect, which obstruct its use in the LPIS


Factors affecting the ETS performance

• Type of reference parcels and structure of the LPIS modelETS more complicated for some types of LPISATS could provide important information for ETS

•Quality of the input orthoimageryPoor image quality will jeopardize the whole ETSAdditional data or processing might be needed

•Operator experience and skillsGood knowledge and experience in land cover mappingSpecial training might be necessary (CAP context, large-

scale mapping)


•Mapping instruction for landscape features•Decision rules for the land cover features in the range of 0.01 and 0.1 ha?

•To map them or not?

•Link between ATS and ETS: ETS could use information from ATS •Refine Land cover classes, according to FAO LCCS•Elaboration of similar ISO compliant quality measures to assess the orthoimagery•Evaluation of the suitability of the VHR data from CwRS to be used for the ETS•AOB: MMU, visual scale, vectorization, …

Future Challenges


Continued by Pieter Roggemans


Status of the ETS pilots

659125026.08.2009MSLPIS 4

80080018.09.2009MSLPIS 3

32780022.09.2009JRCLPIS 2

85080025.08.2009MSLPIS 1

Reference parcels inspected

Minimum number of Reference parcels to check

Date of the pre-selection sample sent from JRC

Work done by

LPIS sample


ETS-scoreboard (partial and provisional)

2.42021.10.124. Proportion defective RP0.001.389.170.12>50%

0.463.974.590.1220-50%

0.464.155.500.4712-20%

1.523.807.950.128-12%

2.8811.2316.820.474-8%

6.2214.6820.491.292-4%

88.4760.7935.4797.410-2%3. Distribution impure RP8.221.7554.71.65>3%2. Proportion impure RP

101.497.2494.6699.761. LPIS eligible area rateLPIS4LPIS3 LPIS2LPIS1%


First feedback from the ETS pilots•ETS might require more efforts than expected,

boundary accuracy and landscape elements considered complex measures

•Parcels found to have critical defects, are not necessary erroneous

Additional analysis of these cases in the national context is necessary

•The land cover mapping in that large scale is not a straightforward process

Specific mapping rules needed to be defined, especially for landscape features. National CAP rules should be considered.

• VHR Orthoimagery might not be sufficient source for ETScertain land cover features are problematic to maptime factor needs to be considered when archive data is used


Conclusions on ETS•Provides lot of objective information on the status of LPIS•Quality indicators defined seems to be meaningful and allows comprehensive analysis•Feasible to be done in relatively short time frame•Applicable in any type of LPIS •Compliant with International Standards (ISO, LCCS, UML)


Thank you for your attention!

executive test suite explained: belgium - flanders...

Documents