input data quality report - british columbiaa100.gov.bc.ca/.../documents/r1498/pem_4039_idq02... ·...

Input Data Quality Report

Review of Abitibi PEM IDQ Report

Submitted to the Ministry of Sustainable Resource Management

By David E. Moon, CDT Core Decision Technologies Inc.

I

REVIEW OF ABITIBI PEM IDQ REPORT .......................................................................................................... 1

0. EXECUTIVE SUMMARY................................................................................................................................. 1

1. INTRODUCTION................................................................................................................................................ 1 1.1. BACKGROUND ................................................................................................................................................ 1 1.2. RATIONALE FOR THE REVIEW........................................................................................................................ 2 1.3. SCOPE OF THE REVIEW................................................................................................................................... 2 1.4. ORGANIZATION OF THE REVIEW .................................................................................................................... 2

1.4.1. Introduction ........................................................................................................................................... 2 1.4.2. The review proper ................................................................................................................................. 2 1.4.3. Conclusions ........................................................................................................................................... 3

1.5. ELEMENTS OF THE REVIEW............................................................................................................................ 3 1.5.1. Organization and Presentation ............................................................................................................ 3 1.5.2. Field Sampling and Field Data ............................................................................................................ 3 1.5.3. Input Data Layers ................................................................................................................................. 3 1.5.4. Knowledge Base.................................................................................................................................... 7

2. REVIEW................................................................................................................................................................ 7 2.1. ORGANIZATION AND PRESENTATION ............................................................................................................ 7 2.2. FIELD SAMPLING AND FIELD DATA............................................................................................................... 7 2.3. INPUT DATA LAYERS ..................................................................................................................................... 8

2.3.1. TRIM Data............................................................................................................................................. 8 2.3.2. Biogeoclimatic Ecosystem Classifcation ...........................................................................................10 2.3.3. Satellite Imagery..................................................................................................................................12 2.3.4. Terrain Stability Mapping ..................................................................................................................14 2.3.5. Forest Cover........................................................................................................................................16

2.4. KNOWLEDGE BASE AND ALGORITHM REQUIREMENTS..............................................................................18 2.4.1. Entities .................................................................................................................................................18 2.4.2. Knowledge Base Attribute Compilation.............................................................................................18 2.4.3. Belief Matrices ....................................................................................................................................18 2.4.4. Knowledge Base Validation................................................................................................................19

3. CONCLUSIONS ................................................................................................................................................19

1

0. Executive summary

It must be noted that the Input Data Quality standards put forth in the Predictive Eco-system Mapping Inventory Standard were, by design, non-prescriptive and flexible enough for PEM practitioners to meet the intent of the standard without being forced to follow rigid formats, formulas, or procedures. The hope was that the standard would encourage critical evaluation of input data by PEM practitioners and encourage its re-porting in a clear, concise, and complete manner. It is therefore difficult to provide a pass/fail evaluation for IDQ reports.

This said, the Abitibi PEM report does not meet fully the letter or the intent of the PEM Inventory Standard for Input Data Quality Reporting.

1. Documentation of metadata for the input data layers while meeting some of the letter of the standard, was incomplete. Of special concern was that the history or pedigree of Forest Cover map revisions, base map conversion, and retrofit pro-cedures were ignored.

2. Documentation of the attribute extraction/derivation procedures was incomplete. In many cases it was not possible to determine the logic or process of extrac-tion/derivation and it was not possible to evaluate the results.

3. Documentation of quality control and quality assurance was inadequate and while the use of field plots established to support development of the knowledge base as a test for the quality of primary or derived input data was commendable, the lack of documentation on sampling strategy and design, lack of documentation on attribute derivation and extraction procedures, lack of documentation on the evaluation of the nature of mapping entities, map entities (spatial accuracy and thematic resolution and accuracy) limited the utility of the information.

4. Despite relatively high error values in those attributes tested against field data, there was no discussion of the sensitivity of ecosystem prediction to errors in the input data. Failure to provide a simple error analysis identifying the impact of indi-vidual and cumulative attribute errors on predictive error significantly reduces the value of this data.

5. The knowledge base was not documented. The attributes used in the knowledge base were not fully defined, and the interpretive logic of the attributes used in the predictive process were not described.

6. The knowledge base was not evaluated against an independent data set.

The short comings make an adequate evaluation of the quality and application of the input data impossible. Furthermore, the failure to adequately document attribute ex-traction methods, attribute data definitions, and knowledge base structure, values and logic makes assessment of the results impossible and precludes incorporation into a provincial knowledge base.

1

1. Introduction

1.1. Background

The Ministry of Sustainable Resource Management commissioned this report as part of a project to evaluate, against the PEM Inventory Standard, Input Data Quality Re-ports submitted in support of PEM projects. Based on these reviews, the contractor was to prepare and report on a framework and format for future Input Data Quality Reports.

The specification for a PEM standard, against which the IDQ Report is evaluated arose, from four preceding works commissioned by the Terrestrial Ecosystem Map-ping Alternatives Task Force of the Resources Inventory Committee (RIC). These re-ports contained important background information and concepts which for the sake of brevity was not included in the standard but was referenced in the expectation that it would be read. These reports were:

1. Towards the Establishment of a Predictive Ecosystem Mapping Standard: A White Paper, by Keith Jones, R. Keith Jones & Associates; Del Meidinger, BC Ministry of Forests, Research Branch; Dave Clark, BC Ministry of Environment Lands and Parks, Resources Inventory Branch; and Fern Schultz, BC Ministry of Forests, Resources Inventory Branch.

2. Problem Analysis on Data Quality Assessment Issues by Dr. David Moon, CDT–Core Decision Technologies Inc.

3. Situation Analysis for Knowledge-Based Systems by Dr. David Moon, CDT–Core Decision Technologies Inc.

4. Problem Analysis on Reliability, Quality Control and Validation of Predictive Eco-system Mapping (PEM) by Dr. Richard Sims and Jeff Matheson, R.A. Sims & As-sociates.

The standard drew upon four additional reports:

1. Specifications for PEM, version 2.1

2. Mapping entities, draft report

3. Protocol for Quality Assurance and Accuracy Assessment of Ecosystem Maps

4. A Method for Large Scale Biogeoclimatic Mapping in British Columbia.

Because PEM was new and largely untested, the original report was somewhat gen-eral and emphasized documentation of data and procedure rather than prescription. The intent was to ensure that a qualified PEM practitioner would be able to evaluate the quality of the input data and procedures used in the production of a PEM map based on the documentation.

In reality, the standard and its referenced reports proved an intimidating, nebulous, time consuming, and open-ended basis for developing IDQ Reports. The variability in quality and format of the reviewed reports is reflective of this reality.

2

1.2. Rationale for the Review

The review evaluates the submitted Input Data Quality Report in terms of both the let-ter and the intent of the Predictive Ecosystem Mapping Inventory Standard for Input Data Quality Assessment. It evaluates organization, completeness, and adequacy of documentation from the perspective of an inventory specialist evaluating the quality of the input data and the adequacy of the procedures used to produce the PEM. It does not evaluate the adequacy of the report as a contract deliverable and it does not evaluate the quality of the data or procedures used.

The original PEM Inventory Standard was neither prescriptive nor detailed and many of the items in the standard were recommendations rather than requirements. In addi-tion, the original standard had not been tested. Finally, the original standard was not presented or intended as a template for reporting. It is therefore not surprising that submissions to date have been highly variable in terms of both format and complete-ness. The review format has therefore attempted to bring a standard format to the re-view and by extension provide an initial template for future IDQ reports.

1.3. Scope of the Review

The review deals only with input data quality reporting, documentation of input data layer processing, and documentation of the knowledge base. It does not evaluate the adequacy of the data or procedures used, and it does not evaluate the conclusions presented in the report. The review evaluates only whether or not there is sufficient in-formation presented for a knowledgeable PEM practitioner to evaluate the adequacy or efficacy of the input data and approach used.

1.4. Organization of the review

The organization of this review attempts to evaluate and report the elements of the PEM process in the logical order in which they are preformed and represents a devia-tion from the order of presentation in the standard.

1.4.1. Introduction

The introduction to the review presents:

1. Background to the review.

2. Rationale for the review.

3. The organization of the review (this section).

1.4.2. The review proper

The review proper has the following major elements:

1. The organization, presentation, and completeness of the report.

2. The individual input data layers.

3. The derivation/extraction and compilation of knowledge base attributes.

3

4. The knowledge base, and validation of the knowledge base.

1.4.3. Conclusions

A general conclusion as to how well the IDQ report meets the letter and intent of the standard.

1.5. Elements of the Review

1.5.1. Organization and Presentation

The review will evaluate the report for organization and presentation; for ease of use and retrieval of information; for format and presentation of report elements including supporting data, use of tables and figures; and for appropriate summary and conclu-sions (specifically are they present and supported by the content of the report).

1.5.2. Field Sampling and Field Data

The standard has no requirement for field sampling however the intent of the standard requires the following documentation for both third party field data and field data col-lected by the contractor in support of the project:

The development of the knowledge base is based in part on field data where the at-tributes being used in the predictive process are collected at locations where the site-series is known. These sites may be collected by the PEM contractor in support of the project, by a third party in support of non-PEM activities, or the contractor may choose a combination of both. Whatever option is chosen, the IDQ report should include the following documentation.

1. The kind, frequency, and distribution of field samples.

2. The method of sample selection (e.g., random, stratified random, selective/modal, etc.) and if stratified or selective the stratification or selection criteria.

3. Quality control and Quality Assurance protocols applied to the field data.

1.5.3. Input Data Layers

Compilation Metadata

Input map metadata provides basic information to ensure that the nature and limita-tions of input maps are understood before creating PEM input data layers. Most im-portant of these are the original base map, projection, and methods used for compila-tion of the map and the history and nature of changes to the original map. Some con-sultants are unaware that TRIM data is compiled and distributed in Albers projection while forest cover and other maps use Universal Transverse Mercator. If obtained in digital form the difference will be obvious when processing is attempted. However if paper maps are digitized by the contractor and the difference is not known, it is prob-able the map will be digitized using the wrong projection. This difference could pro-duce significant positional discrepancies in larger project areas. The metadata should include the process used to match projections, the magnitude of spatial shifts (rubber

4

sheeting) required during the conversion, the verification of thematic boundary accu-racy (if any) and the nature of any thematic content changes during the update.

Spatial Quality of Input Data

Spatial Data Integrity

Reconciliation to TRIM

The spatial integrity metadata should include the method used to match projections between input maps or confirmation of that the conversion was done. The standard established a procedure for evaluating the consistency of the input data map with TRIM features, particularly hydrography. The standard is inadequate with any maps “retrofitted” to TRIM features (e.g., forest cover) unless the procedures and degree of spatial adjustment required to get conformance of TRIM features is recorded.

Spatial integrity:

The standard requires a measure of error for lines which fail to join at map sheet boundaries and for label consistency for polygons crossing map sheet boundaries.

Spatial Accuracy of thematic boundaries:

A more appropriate measure of spatial data integrity is the measurement of geo-graphic coordinates for non-TRIM features such as cultural features which are com-mon to both maps (e.g., cutblocks and roads).

Thematic Data Input Quality

The standard requires that for all input data layers, the suitability of the input data be evaluated for use in the PEM project. Only the spatial accuracy of TRIM data may be taken as given and even this refers only to TRIM features (including digital elevation points). It does not refer to thematic accuracy or to the interpolation of or derivation of landscape attributes such as slope, aspect, shape, slope position et cetera from TRIM data. The resolution of the TRIM data may not be able to adequately portray the scale of landscape characteristics important to the PEM process. This will be especially true in low relief and complex terrain such as hummocky kame and kettle topography, gla-cial fluvial deposits and others.

Thematic Accuracy

This element, while not required, refers to the validation of thematic attributes by com-paring attributes from know geospatial coordinates to those attributes displayed at those coordinates on the map.

Map Entity Suitability

Mapping Concepts

The standard requires an evaluation for appropriateness of and interpretive issues re-lated to the nature of the mapping entities (things mapped e.g., terrain units) and the nature of the map entities (delineations on the map e.g., simple versus complex composition). The review will look for demonstrated understanding of the nature of the

5

mapping concepts used in the input data layers, for an understanding of the interpre-tive issues related to their use for predictive ecosystem mapping, and for documenta-tion of how the issues were dealt with. Specifically, the review will look for documenta-tion evaluating the resolution/complexity of the mapping and map entities, boundary precision for thematic maps, and boundary accuracy for thematic maps. While not specifically required in the standard, these issues are relevant to predictive ecosystem mapping and should be evaluated by competent practitioners.

Resolution/Complexity In the case of thematic maps, resolution refers to the level of detail attached to the mapping entities used in the input data layer and complexity refers to the number and type of mapping individuals used as components in the map entity. With digital eleva-tion data, resolution refers to the precision of the elevation measurement and the spacing of the elevation data points.

Boundary Precision Boundary precision refers the sharpness of the transition between adjacent map units and/or the confidence with which the mapper has drawn the line. Both will influence the accuracy, precision, and interpretation of any spatial overlay product.

Boundary Accuracy Boundary accuracy refers to how closely the delineation of a boundary on the map corresponds to its true location on the ground.

Quality Control

The PEM standard did not require reference to Standard Operating Procedures (SOPs) or quality control protocols although elements of QC protocols are inherent in some of the metadata requirements. Despite this, the intent of the PEM Inventory Standard would be best met with a combination of SOPs and quality control proce-dures to ensure that the SOPs were applied appropriately.

Standard Operating Procedures

Standard operating procedures refer to procedures that have been extensively docu-mented and tested to ensure that they produce consistent, reliable results when fol-lowed. They may be widely accepted and adopted, as with ISO standards, or they may simply be internal procedures that have been well documented and tested. The principal behind a SOP is that once tested and documented, it is only necessary to confirm that the procedure was followed and quality assurance testing can be signifi-cantly reduced.

Quality Control Protocols

Quality control protocols refer to application of standard procedures used to ensure the quality and integrity of either data or products, or it can refer to a series of quality assurance procedures to ensure that acceptable levels of quality are being met. In the case of SOPs, the procedure is assumed to produce acceptable quality and the intent is to ensure that the standard procedures have been applied. Examples include pro-cedures to ensure that field data is correctly transferred to digital format or colour map edits to ensure that thematic data is correctly attached to polygons. Quality control protocols should consist of defined procedures for the production or editing of data, in-

6

formation, or intermediate or final products and a formal sign-off to confirm that the procedure was implemented as documented.

Quality Assurance

Unlike quality control, quality assurance consists of actual testing of the interim and fi-nal products to ensure that acceptable levels of accuracy or reliability are being achieved. When coupled with standard operating procedures, the level and frequency of quality assurance can be significantly reduced.

Knowledge base validation was the only Quality Assurance procedure required for In-put Data Quality Reporting.

Meta Data

The PEM standard establishes minimum levels of documentation and meta-data re-quired to evaluate the quality of input data, predictive procedures, and output products of PEM. The meta-data specified below meet three needs.

1. They provide sufficient information about the nature of the input entities, input data, predictive procedures, and output products for a qualified PEM practitioner to understand the limitations of these items for PEM applications.

2. Their compilation by the PEM practitioner ensures that the practitioner has re-searched the input data and adequately documented the procedures and output products.

3. A longer-term goal of the PEM standard is the eventual integration of PEM/TEM data, information, and knowledge into a single logical data model and repository. The task of integrating TEM with PEM is beyond the scope of this standard but this section will provide the documentation and meta-data necessary to construct such a repository.

Meta data are required in the following areas.

1. Input map source, base, compilation, and map entities.

2. Input map processing including attribute extraction/derivation.

3. Knowledge base and knowledge processing algorithms.

Thematic Compilation and Derivation

Attribute Collection/Derivation and Compilation

The derivation and/or compilation of thematic data layers requires the implementation of a set of procedures to either the original data capture and compilation or the evaluation and processing of third party input data. The standard requires documenta-tion of these procedures either by reference to appended descriptions or to published documents.

7

Wherever possible these references should be to tested Standard Operating Proce-dures described above and should reference the quality control procedures used to confirm the quality of data or the application of tested Standard Operating Procedures.

Attribute Definition

The standard requires documentation of the attributes used in the PEM process. Ele-ments of this documentation are its definition, domain, scale, and units of measure. The standard provides detailed specifications that should be followed.

1.5.4. Knowledge Base

Documentation of the knowledge base requires the following.

1. The PEM entities being predicted.

2. The attributes being used to predict the entities.

3. The method of compilation of attributes for the spatial entities being predicted.

4. The logic, values, and algorithms used to make the prediction.

The standard provides detailed specifications for documentation of the knowledge base.

2. Review

2.1. Organization and Presentation

The organization is simple and straight forward. It provides a very brief introduction and two major sections, Input Data Spatial Assessment and Input Data Thematic As-sessment. Tables are inserted at appropriate locations in the text but appendixes re-ferred to in the text were not submitted with the report nor were referenced materials on methods and procedures. Additional, useful materials were submitted as unrefer-enced spreadsheets.

2.2. Field Sampling and Field Data

The report identified that 650 combined full ecosystem, ground inspection, and visual inspection plots collected following Field Manual for Describing Terrestrial Ecosys-tems, 1998. Although referred to, no sampling plan or strategy was presented. The reader cannot therefore evaluate the method of sample selection or the kind, fre-quency, and distribution of samples. There was no discussion of third party field plots. As a result, the reader has no information on to support that the contractor was com-petent to evaluate the data used, that it was evaluated, or that the data and evaluation were adequate.

8

2.3. Input Data Layers

2.3.1. TRIM Data

Compilation Meta Data

Citation – Not provided.

Consultant/Department – Provided

Compilation Scale - Provided

Publication scale – NA

Period of compilation – Not provided

Original base map and projection used for compilation – not provided

Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided

Current base map and projection used for compilation – not provided

Projection – Not provided.

Mapping entities – NA

Map entities – NA.

Entity relationships – NA


Reconcilliation to TRIM Base

Not Applicable.

Spatial Integrity (input processing)

Edge matching – NA

Positional Accuracy of Thematic Boundaries

NA

Thematic Quality of Input Data

Thematic Accuracy

NA

9


Resolution/Complexity The report makes reference to the resolution of data points being between 45 and 70 m and notes the probable loss of topographic features less than 1 ha in extent. There is no formal evaluation of resolution appropriate to landscape but the contractor did test TRIM derived landscape attributes against field plots.

Accuracy/Precision There was no discussion of the suitability of the accuracy or precision of TRIM data for derived attributes. Only the final attribute classes were tested against field data.

Quality Control

Standard Operating Procedures – NA

Quality Control Protocols – NA

Quality Assurance


The information included in this section is essential to the creation of a provincial knowledge base.

Attribute Extraction/Derivation

Quality Control / Quality Assurance

Standard operating procedures - none referenced

Quality Control Protocols – none referenced

In-house Procedures Documentation – none provided

Quality Assurance The contractor tested derived attributes against ground data. Classes (specified ranges of attribute values) of TRIM derived attributes were tested against field plots. The following information is required to fully evaluate the test.

Sampling Strategy/Plan – no strategy or plan provided.

Accuracy/precision of field plot location – none provided.

Sensitivity Analysis – no sensitivity of interpretive error to data error was reported.

Attribute Definitions

[24]. Definition and description – Description provided but no data definition.

[25]. Attribute code – Code provided

[26]. Method – No method of determination provided

10

[27]. Unit of measure – provided where appropriate

[28]. Entity/Relationship described –The report does not define whether the attrib-ute describes the average for the polygon, dominant value of the polygon, or the value of the dominant component of the polygon.

Numeric Data (real or continuous variable data)

[29]. Precision – not provided

[30]. Statistic – not provided

Categorical Data (discontinuous and unranked)

[31]. Valid values – provided

Ordered Classes

[32]. Rank and limits – rank provided

2.3.2. Biogeoclimatic Ecosystem Classifcation


Citation – None.

Consultant/Department – provided

Publication scale – provided.

Compilation scale – not provided

Period of compilation – not provided


Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided

Current base map and projection used for compilation– not provided

Projection – not provided

Mapping entities – not provided

Map entities – not provided.

Entity relationships – not provided

11



The IDQ report does not make clear whether any existing BEC maps were actually incorporated into localized BEC coverage. It appears that the project used two 1:50K MoF BEC map sheets (93N and 94C) and that the remainder of the area was mapped using rule sets applied to digital elevation data.

[18]. Adjusted control feature shift – not reported.


Edge matching – not reported.

[14]. Edge matching error – not reported

[15]. Attribute/Label matching – not reported

[16]. Raster size – NA.


Not reported


Thematic Accuracy

Discussed in general terms but there are no reported tests against the available field data.


Mapping Concepts – not discussed

Resolution/Complexity The contractor offers an opinion but it is not supported by data or reference.

Boundary Precision Not discussed

Boundary Accuracy Unsupported opinion of the contractor is offered..

12

Quality Control

Standard Operating Procedures – none reported

Quality Control Protocols - None are reported

Quality Assurance

None is reported




[21]. Attribute derivataion – The report refers to “Final Methods Report for Localizing Biogeoclimatic Linework for the Aiken, Tenakihi, Upper Osilinka, Lower Osilinka, and Swannel Landscape Units”, Atticus 2000) but this report was not appended or avail-able to the reviewer.

2.3.3. Satellite Imagery


Citation – None.


Publication scale – NA.

Period of compilation – provided

Original base map and projection used for compilation - NA

Modification History including base map and projection conversions and the basis, method, and nature of thematic updates - NA

Base map and projection used for compilation – Registered to TRIM

Mapping entities – NA.

Map entities – Na.

Entity relationships – NA.



[18]. Adjusted control feature shift – not reported.

13


Edge matching – not reported.

[14]. Edge matching error – not reported.

[15]. Attribute/Label matching – not reported.

[16]. Raster size – not reported.


Thematic Accuracy

None reported despite availability of field data.


Resolution/Complexity Limits to classification including size and type of unit discriminated are discussed in general terms. The nature of the resulting mapping entities are discussed in general terms.

Boundary Precision Issues of probable boundary precision are not discussed..

Boundary Accuracy Issues of probable classification boundary accuracy are not discussed..

Quality Control

Standard Operating Procedures – none are reported

Quality Control Protocols – none are reported.

Quality Assurance

None is reported.




[21]. Attribute extraction – no methods are reported, referenced, or discussed for the classification of image data into vegetation types..

14


[24]. Definition and description – none provided except a brief description of the code.

[25]. Attribute code – provided

[26]. Method of Measurement – NA

[27]. Unit of measure – NA

[28]. Entity/Relationship described – a brief description of the map units identifies the vegetation class as a complex of broad vegetation types.


[31]. Valid values – provided

2.3.4. Terrain Stability Mapping


Citation – none provided


Publication scale – Terrain Survey Intensity Level is reported

Period of compilation – provided


Modification History including base map and projection conversions and the basis, method, and nature of thematic updates - not provided

Current base map and projection – Not provided

Mapping entities – provincial Terrain Standard is referenced.

Map entities – provincial Terrain Standard is referenced..

Entity relationships – Not porvided



[18]. Adjusted control feature shift – not reported


Edge matching

15

[14]. Edge matching error – not reported

[15]. Attribute/Label matching – not reported



Not reported


Thematic Accuracy

Terrain stability maps were correlated by provincial authority no independent evalua-tion was done despite the availability of 650 geo-referenced plots.


Mapping Concepts

No demonstrated understanding of the nature of Terrain Stability map units or the im-plications of map unit concepts for PEM production.

Resolution/Complexity Not discussed or reported.

Boundary Precision Not discussed or reported..

Boundary Accuracy Not discussed or reported

Quality Control

Standard Operating Procedures –

The work is completed in accordance with the Standards for Digital Terrain Data Cap-ture in British Columbia (June 1998), and the Guidelines and Standards for Terrain Mapping in British Columbia (RIC, 1996), and Terrain Classification System for British Columbia (Howes and Kenk 1997)

Quality Control Protocols

The survey intensity level which includes frequency of ground inspection is reported for each Terrain Stability Map sheet used but there is no report of ground inspection procedures.

16

Quality Assurance

The contractor provided no independent quality assurance despite the availability of 650 ground inspection forms.




[21]. Attribute extraction – the report identifies neither specific terrain features used nor their method of extraction for use in the knowledge tables.


Attributes extracted from Terrain Stability maps are neither reported nor defined.

2.3.5. Forest Cover


Citation – Specify a reference to a formal, published source of the data, if available.

Consultant/Department – Specify the public or private sector organization(s) re-sponsible for collecting, compiling, and maintaining the data and an appropriate con-tact within the organization(s).

Compilation scale – not provided

Publication scale – Specify the original publication scale of the inventory.

Period of compilation – not provided


Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided.

Current base map and projection – provided

Mapping entities – not provided.

Map entities – not provided.

Entity relationships – not provided.



[18]. Adjusted control feature shift – not provided.

17


Edge matching

[14]. Edge matching error – not provided

[15]. Attribute/Label matching – provided



Addressed in general terms


Thematic Accuracy

Leading species, secondary species, height class, and age class were tested using a confusion matrix against field plots. Only brief summary tables were available in the report. Details of the “confusion” matrix were submitted as unreferenced spreadsheets after additional information was requiested.


Mapping Concepts – not discussed

Resolution/Complexity – not discussed

Boundary Precision – not discussed

Boundary Accuracy – not discussed

Quality Control

Standard Operating Procedures – none provided

Quality Control Protocols – none provided

Quality Assurance

Six hundred and fifty ground samples were collected to provide training data for the knowledge base and for testing the accuracy of thematic information in the input maps. Leading species, secondary species, height class, and age class were tested using a confusion matrix against field plots. The information identified below is neces-sary to evaluate the efficacy of quality assurance procedure.

Sampling Strategy/Plan – no strategy or plan provided.

Accuracy/precision of field plot location – none provided.

Sensitivity Analysis – no sensitivity of interpretive error to data error was reported.

18




[21]. Attribute extraction – method not provided.


[24]. Definition and description – not provided

[25]. Attribute code – provided

[26]. Method – not provided.

[27]. Unit of measure – not provided

[28]. Entity/Relationship described – not provided

Numeric Data (real or continuous variable data)

[29]. Precision – not provided.

[30]. Statistic – not provided


[31]. Valid values provided

Ordered Classes

[32]. Rank and limits - provided

2.4. Knowledge Base and Algorithm Requirements

2.4.1. Entities

Entity definitions were provided as an unreferenced spreadsheet. The spread sheet lacked both annotation and a data dictionary.

2.4.2. Knowledge Base Attribute Compilation

[19]. Mapping entity cross product correlation – No information provided

[20]. Sliver adjustment – No information provided.

2.4.3. Belief Matrices

No information provided.

Attribute Values and Condition Sets

19

No information provided

2.4.4. Knowledge Base Validation

No information provided.

3. Conclusions

This report does not meet fully the letter nor the intent of the PEM Inventory Standard for Input Data Quality Reporting.

1. Documentation of metadata for the input data layers was incomplete. Of special concern was that the history or pedigree of Forest Cover map revisions, base map conversion, and retrofit procedures were ignored. Also of concern was that there was no explicit recognition of projection discrepancies between TRIM and other input data layers.

2. Documentation of the attribute extraction/derivation procedures was incomplete.

3. Documentation of quality control and quality assurance was inadequate and while the use of field plots established to support development of the knowledge base as a test for the quality of primary or derived input data was commendable, failure to document sampling strategy and design, failure to document attribute deriva-tion and extraction procedures, failure to evaluate the nature of mapping entities, map entities (spatial accuracy and thematic resolution and accuracy) limited the utility of the information.

4. Despite relatively high error values in those attributes tested against field data, there was no discussion of the sensitivity of ecosystem prediction to errors in the input data. Failure to provide a simple error analysis identifying the impact of indi-vidual and cumulative attribute errors on predictive error significantly reduces the value of this data.

5. The knowledge base was not documented. The attributes used in the knowledge base were not fully defined, the interpretive logic of the attributes used in the pre-dictive process were not described, and the nature of the mapping and map units being predicted were neither defined nor described.

6. The knowledge base was not evaluated against an independent data set.

The short comings make evaluation of the quality and application of the input data im-possible. Furthermore, the failure to adequately document attribute extraction meth-ods, attribute data definitions, and knowledge base structure, values and logic makes assessment of the results impossible and precludes incorporation into a provincial knowledge base.

input data quality report - british columbiaa100.gov.bc.ca/.../documents/r1498/pem_4039_idq02... ·...

Documents