Building a User-Driven GEOSS: Methods to Capture, Analyze, and Prioritize User Needs

GEO Task US-09-01a and Earth Observations Identification

Lawrence Friedl, USA-NASA User Interface Committee Member

US-09-01a Task Lead

ISRSE-34 Symposium • Sydney, Australia10.April .2011

Earth Observations NeedsOverview of Presentation Elements

This presentation discusses a document-based approach to identify observation needs, using teams of experts to:

a) review analytic methods to assess needs in documents, and b) concur on priority-setting criteria to rank the needs.

Based on results of GEO Task US-09-01a: Identifying Critical Earth Observation Priorities.

I. Approach and Meta-analysis

II. Results

III. Findings

IV. Advantages and Limitations

Discussion 2

Earth Observations Needs

Section I.

Approach and Meta-analysis

3

Earth Observations Needs Task US-09-01a

GEO Task US-09-01a

Objective:Establish and conduct a process for identifying critical Earth observation priorities common to many GEO societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes.

GEO Societal Benefit Areas (SBA):Agriculture Disasters HealthBiodiversity Ecosystems WaterClimate Energy Weather

4

Earth Observations Needs Task US-09-01a

GEO Task US-09-01a

Objective:Establish and conduct a process for identifying critical Earth observation priorities common to many GEO societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes.

GEO Societal Benefit Areas (SBA):Agriculture Disasters HealthBiodiversity Ecosystems WaterClimate Energy Weather

5

Earth Observations Needs Task US-09-01a

US-09-01a Process: Nine Steps The process lists the steps serially, yet some are done in parallel.

Step 1: UIC Members identify Advisory Groups and Analysts for each SBA

Step 2: Determine scope of topics for the current priority-setting activity

Step 3: Identify existing documents regarding observation priorities for the SBA

Step 4: Develop analytic methods and priority-setting criteria

Step 5: Review and analyze documents for priority Earth observations needs

Step 6: Combine the information & develop a preliminary report on the priorities

Step 7: Gather feedback on the preliminary report

Step 8: Perform any additional analysis

Step 9: Complete the final report on Earth observations for the SBA

6

Earth Observations Needs Task US-09-01a

General Approach

Document-based analysis. Examined observation needs expressed in publicly-available documents from past ~10 years. Analysis included over 1700 documents.

An “Advisory Group” and an “Analyst” worked to identify documents, analyze them, and prioritize observations within each SBA. AGs involved 6-23 people from developed & developing countries that represent experts in an SBA. (Invited CoPs, IGOS, others to be on AGs). 167 AG members & 43 countries in total.

Developed individual SBA reports, which specified the observation priorities for that SBA.

7

Based on the SBA reports, the Task Team conducted a meta-analysis across the individual SBA reports to determine observations priorities common to many SBAs.

The team combined and prioritized parameters from the SBA lists and prepared an over-arching report to identify “Earth observation priorities common to many SBAs.”

The report includes findings, lessons learned, and recommendations.

Earth Observations Needs Task US-09-01a

Cross-SBA Analytic Methodologies

Method 1: Tally of All PrioritiesFrequency analysis is a simple tally of the SBAs that require a given observation. (Total of 146 observations were included in this prioritization.) Methods 2&3: Weighted Sums of All PrioritiesWeighted frequency analysis is a weighted sum of the number of SBAs that require a given observation, taking into account the high/medium/low importance assigned by SBA Analysts. Different weighting schemes in the two methods (Same 146 parameters as Method 1.)

Method 4: Top 15 Priorities by SBAThis key parameters method is based on each SBA Analyst preparing a list of the “top 15” for that SBA. (Total of 99 observations were included.)

Final Set: Ensemble approach across the methods. Calculated mean rank and the range of ranks for all 146 parameters across methods. Natural breaks at top 19 and top 36 parameters.

8

Earth Observations Needs

Section II.

Results

9

Earth Observations Needs Task US-09-01a

SBA-specific Observation Priorities

Each SBA identified a set of observation priorities as well as the 15 “most-critical” observations. Overall, there were 146 observations identified as priority needs from combining the SBA lists. The Task Team used an ensemble of 4 prioritization methods to integrate and prioritize for the Cross-SBA Analysis.

Agriculture SBA Disasters SBA Energy SBA

10

Earth Observations Needs Task US-09-01a

8 observations (5% of 146 total) are common to 6

or more SBAs

29 observations (20%) are

common to 4 or more SBAs

100 observations (68%) are

common to 2 or more SBAs

11

GEO Task US-09-01aHighest Ranked Observations (#1-20) and Associated SBAs

All of the 20 Highest-ranked observations are common to 4 or more SBAs

All of 30 Highest-ranked obs. are common to 3 or more SBAs

3 Highest-Ranked: Precipitation, Soil Moisture, Surface Air Temperature

* Biodiversity SBA Team did not produce a set of Earth observations priorities.

Note: Some observations may be relevant to an SBA even though not included in the SBA’s set of priorities.

12

Earth Observations Needs

Section III.

Findings

13

Earth Observations Needs Task US-09-01a

Task Report – Findings

•Precipitation Reigns the Cross-SBA Analysis

•Methods Showed Agreement at Highest-Rankings

•Priorities of a Single SBA May Not Be on the Cross-SBA List

•Task’s Approach Produced Users’ Needs in Users’ Terminology

•Articulation of Observation Needs in Documents Varied

•Regional Needs Incorporated but Not Featured

•Availability of Documents by Region Varied

•Insufficient Information across Documents on Parameter Characteristics

14

Earth Observations Needs Task US-09-01a

Task Report – Findings

•Priorities of a Single SBA May Not Be on the Cross-SBA List

This approach, by design, focused on the commonality of priority observations to many SBAs. Thus, some observations of critical importance to a particular SBA do not appear in the Cross-SBA list of priority observations.

•Task’s Approach Produced Users’ Needs in Users’ Terminology

Users didn’t always use terms that the Earth obs. community may be familiar with. Many of the needed observation parameters were expressed as phenomena of interest rather than technical specifications of the parameter. Overall, the demand-side, user-based approach of the task produced a rich array of observations and revealed a need for follow-on user engagement to refine parameter characteristics.

•Insufficient Information across Documents on Parameter Characteristics

Some of the documents included quantitative information on the required characteristics (e.g., accuracy, latency) of critical observation parameters, while other documents lacked such information. At times it was available. However, the required observation parameter characteristics vary widely according to the user and application.

15

Earth Observations Needs Task US-09-01a

Task Report – Recommendations

•Gather information and engage users on specific characteristics of the priority Earth observations, especially Precipitation.•Conduct an assessment of the current and planned availability of the priority Earth observations.•GEO and/or Regional Caucuses could consider pursuing similar assessments at regional levels.•Consider additional analytic methods to gathering users’ needs and pursue an ensemble of approaches.•Prescribe the prioritization methods, SBA sub-areas, and other aspects of the SBA analyses.•Pursue broader incorporation of documents in many languages.•Continue the use of ad hoc Advisory Groups, with refinements.•Strongly consider a single organization to manage the individual SBA analyses.•Articulate an SBA’s community of users to support systematic collection of users’ needs.

16

Fo

llow

-on

A

ctivitiesP

rocess

Imp

rovem

ents

Earth Observations Needs Task US-09-01a

Task Report – Recommendations

•GEO and/or Regional Caucuses could consider pursuing similar assessments at regional levels.

User needs vary by geographic region, and user needs unique to a single region were not likely to appear among highly-ranked observations of the Cross-SBA analysis

•Pursue broader incorporation of documents in many languages.Documents describing user needs, especially regional and national needs, exist in languages other than English, yet such documents were not discovered or were underrepresented. The Task Team recommends that future endeavors plan and provide sufficient resources for the identification of documents in many languages and for necessary translations

•Articulate an SBA’s community of users to support collection of users’ needs.Users and end-users are broad terms. Future Task Teams should develop or refine a set of User Types for each SBA. The Task Teams should ensure that the Analysts employ the User Types as guidance in collecting information and representing needs, utilizing them to conduct a gap analysis of their activities to address any bias or gaps.

17

Earth Observations Needs

Section IV.

Advantages and Limitations

18

Earth Observations Needs Task US-09-01a

Task Lead’s CommentsTask’s efforts and results represent significant contributions and first steps within GEO to articulate Earth observation priorities. GEO has documented in a transparent way how Earth observation needs have been identified, involving numerous organizations and experts. The value of the Task’s results and cross-SBA report is at least twofold:

• Provide a baseline and entry point for further engagement with end users on their needs.

• Confirm any expected priority observations as backed up by an analysis of the literature.

19

Earth Observations Needs Task US-09-01a

Lessons Learned•Task Approach Achieved Desired Diversity in Prioritization Methods

•Variety in Analysts’ Approaches Introduced Complexities

•Approach to Sponsorship of Analysts Impacted Process

•Approach to Selection of SBA Sub-Areas Introduced Challenges

•Advisory Groups Played Valuable Yet Variable Roles

Use of existing documents provides a level of objectivity

The ranking of an observation in the Cross-SBA list does not imply objective importance of that observation as much as commonality in need.

20

Earth Observations Needs Task US-09-01a

A Document-Based Approach

Process Comments• Analytic methods to extract observations and needs

- Extraction and inference

• Prioritization methods and priority-setting criteria- Opportunity to tailor to SBA-specific issues (e.g., Daily Adjusted Life Years, World Energy Outlook)

- For individual analyses, need to be internally consistent- For meta-analyses, need commonality and comparability across types (e.g., ordered list, tiers, grouped by some factor)

Note: The US-09-01a approach, by design and broadly-stated, was to be a natural experiment involving a variety of methods; methods were tied to the respective SBA communities as represented by the AGs.

21

Earth Observations Needs Task US-09-01a

A Document-Based Approach

Assumptions• Documents were produced in an unbiased way• Documents express needs for existing and new observations

22

Earth Observations Needs Task US-09-01a

A Document-Based Approach

Advantages•Independent and objective (based on what organizations have already stated publicly) •Reflects collective needs of organizations across multiple people•Shows documented, expressed needs•Based on a ‘democratic’ principle in which all were eligible to be considered•Generate observation needs in the terms and lexicon of the respective user communities

23

Earth Observations Needs Task US-09-01a

A Document-Based Approach

Limitations•Difficult to capture very-recent needs (the time aspect of the approach features long-term, sustained expressions of need); •Advisory Group member engagement and potential bias •Need for representative set of documents for all regions and SBAs•Level of specificity and technical information on needs may vary across documents; inference of needs introduces possible error

24

Earth Observations Needs Task US-09-01a

A Document-Based Approach

25

p. 51

Earth Observations Needs Task US-09-01a

Recommendations•Document-based analysis provides an objective basis to an identification of Earth observation needs

– should be part of a ensemble of analytic & prioritization methods

– should be employed to identify users’ terminology for their needs

•A process involving a document-based analysis across themes should prescribe the analytic/prioritization method(s) and deliverables as well as allow for SBA-specific variants

•Allocate significant time, resources, etc. to include range of documents across regions, languages

•Articulate what a ‘representative’ sample comprises

– should use and internal gap analysis to ensure appropriate breadth

•Allocate sufficient time for review of preliminary results by people outside of advisory groups; this review may stimulate the identification of a host of new documents)

26

Earth Observations Needs Task US-09-01a

Contacts & Acknowledgements

Lawrence Friedl, US-09-01a Task LeadAmy Jo SWANSON, Task CoordinatorErica ZELL, Lead AnalystAdam CARPENTER, Asst. Lead Analyst

Special recognition to the Analysts and Advisory Group members for the respective SBAs.

Final Cross-SBA Report and Individual SBA Reports are available at the US-09-01a Task Website:

http://sbageotask.larc.nasa.gov/27

Earth Observations Needs

Discussion

28

Earth Observations Needs

Back-up Materials

29

Earth Observations Needs Task US-09-01a

Advisory Groups for SBAs

30

• Advisory Groups ranged from 6-23 members.• Sought broad international and regional representatives.• Involved GEO CoPs, former IGOS Themes, GEO Countries

and GEO Participating Organizations.

Earth Observations Needs Task US-09-01a

This chart presents the 30 highest-ranked Earth observations, shown according to score in the Cross-SBA analysis. The range in ranks is also shown.

Ranks are ‘inverted’ so highest score is 146.

31

Earth Observations Needs Task US-09-01a

General structure: An “Advisory Group” and an “Analyst” work together to identify

documents, analyze them, and prioritize observations within each SBA.

Advisory Group (1 per SBA)

Functions:- Will help to identify documents- Comment on analytic methods

and priority-setting criteria- Review the analysts’ findings,

priorities, and reports.

Involved 6-23 people from developed and developing countries that represent experts in an SBA.

Involved Communities of Practice, former IGOS Themes, GEO Countries and Participating Organizations.

Analyst (1 per SBA)

Functions: - Will read and analyze the documents- Develop an analytic method and priority-setting criteria- Conduct the meta-analysis to identify common priorities within a SBA.

The Analyst was the primary coordinator and organizer of the activity to meet the schedules and deadlines.

Interacted with and utilized the Advisory Group to vet prioritization methods and review results and reports. 32

Earth Observations Needs Task US-09-01a

Advisory Groups

33

Earth Observations Needs Task US-09-01a

A Document-Based Approach

34

Earth Observations Needs Task US-09-01a

Final Report

Published October 2010.

Delivered to GEO Plenary VII as annex to UIC Report and distributed at Plenary.

Final Cross-SBA Report and Individual SBA Reports are available at the US-09-01a Task Website: http://sbageotask.larc.nasa.gov/

35