Exploratory Analysis of Forestry Data in NEFIS

Natalia Andrienko & Gennady AndrienkoFHG AIS (Fraunhofer Institute for Autonomous

Intelligent Systems)http://www.ais.fraunhofer.de/and

NEFIS Project Workshop, JRC Italy, 29th June 2005

NEFIS and our research

• Our research focus is EDA – Exploratory Data Analysis (in particular, spatial and temporal data)

• In NEFIS, we strive at explaining and promoting the ideas and principles of EDA

• We have used the ICP Forests defoliation data as a non-trivial example to demonstrate systematic, comprehensive EDA

• We hope to receive valuable feedback from you for guiding our further work

What Is EDA?

• Emerged in statistics in 1970ies; originator: John Tukey

• A philosophy and discipline of unbiased looking at data: “What can data tell me?” rather than “Do they agree with my expectations?”– Similar to the work of a detective (J.Tukey)

• Need to look at data focus on visualisation and user interaction with data displays

Purposes of EDA

• Uncover peculiarities of the data and, on this basis, understand how the data should be further processed (e.g. filtered, transformed, split into parts, fused, …)

• Generate hypotheses for further testing (e.g. using statistical methods)

• Choose proper methods for in-depth analysis (possibly, domain-specific)

• Especially important for previously unknown data, e.g. found in the Web relevant to NEFIS

EDA vs. other analyses

• EDA does not substitute rigor methods of numerical analysis, either general or domain-specific, but should give the understanding what methods and how to apply

Original data

1. EDAUnderstanding

of the data (mental model)

2. Data processing

Processed data

3. In-depth analysis

Conclusions, theories,

decisions, …

EDA vs. information presentation

• EDA makes intensive use of graphics• However, “nice” presentation and reporting are

not EDA purposes• Primary goal of presentation: convey certain

idea or set of ideas to others– Understandably– Convincingly– Aesthetically attractively

• This requires different visual means than exploration

The defoliation data

• Large volume: 6169 spatially-referenced time series

• Two dimensions: S&T• Many missing values• No full compatibility

across countries, species, time etc.

EDA: data quality issues

Specialists’ opinion (after seeing the draft report of the data exploration): “The data were not meant for analysis!”

But:1. There are no ideal data (especially in the Web and

for free)2. Even for understanding data inadequacy one needs

first to explore them3. Even imperfect data can be useful4. The principles of EDA (demonstrated further) are

applicable to perfect data as well

General procedure of the EDA

1. See the whole– Space + Time 2 complementary views

1) Evolution of spatial patterns in time

2) Distribution of temporal behaviours in space

2. Divide and focus– Data are complex Have to be explored by slices

and subsets (species, age groups, countries, years, …)

3. Attend to particulars– Detect outliers, strange behaviours, unexpected

patterns, …

See the whole: Handle large data volumes

• General approach: Data aggregation

• Task 1: Explore evolution of spatial patterns

• Appropriate data transformation: aggregate by small space compartments (regular grid with 4025 cells); separately for different species; various aggregates (mean, max)

Gain: no symbol overlapping

Explore evolution of spatial patterns

a) Animated mapb) Map sequence

Observations:• Persistently high

values in Poland• Improvement in

Belarus• Mosaic distribution in

most countries: great differences between close locations

• Outliers

Divide and Focus: Exploration on country level

• Recommendable due to inconsistencies between countries

• Observation: abrupt changes between locations spatial smoothing methods are not appropriate

Explore spatial distribution of temporal behaviours

• Are behaviours in neighbouring places similar?

• Step 1. Smoothing supports revealing general patterns and disregarding fluctuations and outliers (we shall look at outliers later)

Explore spatial distribution of temporal behaviours

• Are behaviours in neighbouring places similar?

• Step 2. Temporal comparison (e.g. with particular year, mean for a period) helps to disregard absolute differences in values and thus focus on behaviours

Observation: no strong similarity between neighbouring places

Compare behaviours in plots with different main species

• Mosaic signs: – 6 rows for species; – 14 columns for years

1990-2003;– Colours encode

defoliation values

Observation: behaviours differ for different main species

Explore overall temporal trends

Line overlapping obstructs data analysis apply aggregation

Aggregation method 1: by quantiles

Aggregation method 2: by intervals

Divide and Focus: Germany

Divide and Focus: age groups 1,3

Attend to particulars

Types of particulars (examples):– Extreme values– Extreme changes– High variability– …

Questions:– When?– Where?– What is around?– Why? (a question for further, in-depth analysis)

Domain knowledge is essential

Attend to particulars: extreme values

1. Click on a segment corresponding to extreme values

2. The behaviour(s) is(are) highlighted on the time graph

3. The location(s) is(are) highlighted on the map

Attend to particulars: what is around?

• In some neighbouring places the behaviours during the period 2000 - 2003 are somewhat similar

Attend to particulars: extreme changes

1. Transform the time graph to show changes

2. Select extreme changes in a specific year (here 2003)

Attend to particulars: high variation

1. Aggregate time graph by quantiles

2. Save counts

3. Visualise e.g. on a scatter plot

4. Select items with high variation

Attend to particulars: high fluctuation

• Select items with maximal number of jumps between quantiles

Attend to particulars: stable extremes

• Select items being always in the topmost 10%

Attend to particulars: stable increase

1. Turn the time graph in the segmentation mode

2. Choose “increase” and set minimum difference

3. Select a sequence of years by clicking

4. Check sensitivity to the time period!

Conclusions: the Data

• This dataset is not suitable for application of major statistical analysis methods due to– absence of spatial & temporal smoothness– skewed distributions– outliers– missing values

• The data may be suitable for other purposes (e.g. in a context of a broader study of the ecological situation over Europe)– EDA methods can promote insights

Recap: Exploration procedure

• See the whole– Evolution of spatial patterns in time– Distribution of temporal behaviours in space

• Divide and focus– Data were explored by slices and subsets

(species, age groups, countries, years, …)

• Attend to particulars– Extreme values, extreme changes, high

variation, high fluctuations, stable growth …

Recap: Tools

• Visualisation on thematic maps, time graphs, other aspatial displays

• Aggregation: reduce data volume & symbol overlapping

• Filtering: divide and focus (select subsets)• Marking: see corresponding data on different

displays• Data transformation: smoothing, computing

changes, normalisation etc. It is important to use the tools in combination

Further information• Software: http://www.commongis.com

• Scientific issues (papers, tutorials, demos): http://www.ais.fraunhofer.de/and

• Book to appear:

N. and G. Andrienko

“Exploratory Analysis of Spatial and Temporal data. A Systematic Approach”

(Springer-Verlag, end 2005)A systematic approach to defining tasks, tools, and principles of EDA

In press, to appear end 2005

http://www.ais.fraunhofer.de/and