Summary of a joint CEOS-WGCV and GOFC-GOLD workshop on validation of vegetation continuous fields products

Operational land cover observations require commonly accepted validation protocols so that users can understand the limitations of products and so that producers can improve them. Land cover is typically characterized by a classificatory scheme, which assigns each pixel to one of a number of classes. An alternative approach is to characterize each pixel in terms of the proportion occupied by a particular class such as the percentage tree cover, creating so-called vegetation continuous fields (VCF) products. For the evolving suite of VCF products there is only limited accuracy information, and neither are there an integrated sampling framework or field measurement protocol to implement a thorough product validation on regional and global scales.

On 27th and 28th October, a workshop was held at South Dakota State University’s in the Geographic Information Science Center of Excellence in Brookings, South Dakota, to discuss and develop concepts for validating VCF data sets. This meeting was a continuation of the joint activities of the CEOS Working Group for Calibration and Validation (WGCV) and GOFC-GOLD to evolve standards for land characterization and validation. This has already resulted in a document describing best practices for validating conventional classifications of global land cover data sets. Similarly, the overall objective of the VCF validation meeting was to present and discuss approaches to validate continuous vegetation cover estimates and develop consensus from the science community, producers and users of VCF products on the most suitable approaches for their validation and for confidence-building so that such data will have much wider use. The workshop was attended by 22 experts representing a variety of universities, agencies, and user organizations from different parts of the world (e.g. CEOS, GOFC-GOLD, USGS, JRC, INPE, USFS, USDA, etc.). The following sections provide an overview of the workshop discussions and outcomes.

New VCF products

The suite of MODIS VCF products will be extended within the next year. The current single-date 500 m product of tree, herbaceous, and bare cover will be improved to a global 250 m multi-year data set including a larger variety of thematic layers. They include trees, woody shrubs, herbaceous, bare, agriculture, water, as well as, leaf longevity and leaf type. Developing a validation framework for all these upcoming products including change characteristics seems essential; however, the workshop’s technical discussions were largely focused on the assessing single-date tree cover datasets.

Validation objectives

There is more than one reason for validating VCF layers. Robust accuracy assessment is important for the production of maps and to support good science. Many international environmental protocols and agreements require that map products are independently evaluated and can potentially be challenged. User communities often request validation statements but also require interoperability with any data and analysis system in place. Thus, inter-comparisons with existing data in the form of in situ or map products) may also be considered in the accuracy assessment processes to build user confidence in new earth observation products.

Validation techniques

The validation approaches for continuous fields data are somewhat different than for covnetional land cover classificiations. The sampling design has to consider the continuous heterogeneity of the land surface and the different reference data acquisitions and analyses; accuracy reporting will be based on measures like correlation and root mean square error instead of the classical error matrix. In-situ observations require quantitative measures of cover density from representative reference sample areas or transects. Such information needs to be scaled for comparison with big pixels of moderate resolution global map products such as those from MODIS. Different intermediate resolution remotely sensed data may be used for this purpose, e.g. 1-5 m resolution data allow tree crown identifications or Landsat-type data provide fine-resolution fractional cover estimates where there are closed canopies and the landscape is not heavily fragmented. Timing and the spatial support domain of reference data are key considerations since VCF conventionally represents vegetation at peak conditions as sub-pixel estimates.

A suite of available datasets was identified, which may be useful for VCF verification and harmonization purposes. Prominent examples for tree cover are forest inventories, existing in situ datasets and networks (i.e. BIGFOOT, VALERI, GTOS-TEMS, LTER, GLC2000 and MODIS sites), and several other available global and regional land cover map products. However, existing data sets rarely provide reference information thorough enough that they can directly used to validate VCF measurements. Specific cross-walking procedures may need to be developed . Most likely the comparisons will provide qualitative indicators of performance useful for regional-scale confidence building rather than quantitative validation assessment.

Implementation framework

The implementation of a global operational validation requires a joint international initiative. GOFC-GOLD and the CEOS WGCV have outlined such a framework for global land cover datasets. It describes the processes for the establishment and operation of such a framework and defines roles and responsibilities to involve key actors for best use of resources and experiences. This initiative is a response to the GCOS Implementation Plan of the UNFCCC. The VCF validation efforts are expected to join and profit from this ongoing initiative ultimately providing robust, comparative, and updated accuracy assessment for all key global land cover datasets. Until then, studies are encouraged that support better understanding of VCF product accuracy both as quantitative and confidence building validation and verification exercises.

Codifying best practices

The participants of the workshop agreed to summarize the consensus on approaches to VCF validation in a “best practices” document. This effort will follow the model of the “Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps” developed by CEOS WGCV and GOFC-GOLD; a document currently in print.

An outline of the corresponding VCF document was prepared during the workshop and will be developed within the next months based on the workshop findings. This document for validating discrete land cover data sets will draw upon other completed validation efforts notably the afore-mentioned document produced by CEOS WGCV and GOFC-GOLDVCF validation is just starting and is expected to benefit from validation of related variables depicted in similar ways such as LAI and from other related ongoing studies.

Overall, the workshop provided a successful start in this arena. Not all questions raised at the workshop may have been answered properly, but a road map was developed to move forward. At least one additional workshops will be needed.

Martin Herold
Matt Hansen
John Townshend
29.10.05

Outline of VCF Validation Issues

1. Significance of VCF
1. Thematic layers
1. Uses of VCF
2. Sources of uncertainty
3. Issues involving big pixel validation
4. Validation objective
5. Thematic layers
1. User guidance on VCF value
2. Input to improvements to future products
2. Designing a sampling scheme
1. Probabilistic
2. Alternative methods
3. Global schemes
4. Regional schemes
5. Reporting of validation (RMSE etc), multi-scale reporting
3. Confidence building using existing data sets
4. VCF validation using in situ measurements
5. VCF validation using finer-resolution remotely sensed data
6. Integrated in situ and finer-resolution remotely sensed data
7. Considerations for validating VCF change maps
8. Developing institutional partnerships
1. Introduction – Martin’s diagram
2. Linkages with other efforts
3. Challenges in developing and porting a protocol
4. Reporting of results/data sharing
5. Phasing in protocols
6. Relating results to new products

Participants

Agenda and links to presentations

Agenda

Thursday, October 27

Friday, October 28, 2005