From icons to mixed realities: how external representations help us

learn

Yvonne Rogers

SLIS/Informatics

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Overview

• How can we design graphical representations to facilitate learning and understanding?

• How do they work?– Public symbols– Icons– Diagrams– Animations– Multimedia– Mixed realities

Claims about their added value

• Symbols can be universally understood without ever having been seen before

• Icons are easy to infer meaning and remember• Diagrams can make relationships explicit• Animations can convey dynamic changes• Multimedia can ‘enrich’ learning• Mixed realities provoke reflection

My research agenda

• What are the main cognitive benefits of different forms of external representations?

• Can we develop a generalisable theoretical account of how external representations work?

• Can this provide design principles and guidance?

• Can we show whether the theory and principles are valid and useful?

Sometimes a (good) picture can be worth a 1000 words

Sometimes not…

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

1/5 people thought it signified a kite flying area

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

3/5 people though it signified foot-and-mouth disease

And sometimes they are just confusing…

Public symbols

• Can be easily understood

• Can be amusing

• Can be ambiguous

• Context is important

• May need to learn a ‘picture’ grammar

Icons - why use instead of text?

• Many objects and operations can be succinctly expressed in iconic form– e.g., folders, files, trash can, to save, to delete

• Many functions are poorly expressed in an iconic form– e.g., format, preferences, to quit

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Icons

• Can be easily understood and meaning remembered

• Can be easy to scan and locate in a set• Can be confusing if look alike - difficult

to discriminate• Direct mappings best, followed by

symbolic and then abstract

Diagrams

• Why use diagrams instead of text?

Diagrams

• Help us understand difficult and complex subjects (e.g. an ecosystem)

• Make relationships and interdependencies between elements explicit

• Make it easier to ‘read off’ information and see connections

Animated diagrams

• What is the added value?

• Same benefits as static ones but also…

• Can show invisible and dynamic processes

A static and animated diagram of how the heart works

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Is a moving picture worth 1000 static ones?

+ Animations can be effective at displaying dynamic changes

+ They make explicit what is implicit in diagrams

- Difficult to keep track and ‘digest’

- Learners simply watch and assume they have understood

Is Multimedia any better?

• Multimedia typically consists of graphics, text, speech and links

• Provides more representations and more information

An early example of multimedia: Mayo’s Health CD-ROM

text...

search...

...graphics

Multimedia

• Early educational CD-ROMs were designed as electronic encyclopedias

• Knowledge represented in multiple media based on

a resource/book model • Full potential of multimedia not being realized

• How could it be designed to facilitate the learning process?

Designing it to teach the ‘difficult stuff’

An example of ‘difficult stuff’:reasoning about a complex system using an abstraction

Example: What would happen if the mice were wiped out?

To answer need to:

• know factual knowledge (who eats what)• understand canonical links (arrows)• reason about knock on-effects (using representations in temporal and spatial order)

Pondworld multimedia environment

• Goal was to enable children to reason more effectively about ecosystems using food web diagrams

• Get the children to relate explicitly their concrete knowledge with the abstract formalism

• To achieve this by enabling them to manipulate and construct dynamic food web diagrams and to get feedback on their actions

Multimedia can be much more…

• Different elements can be conveyed simultaneously

• Can be interacted with in novel ways

• Studies of children using PondWorld showed significant improvement in ability to reason using diagram

External cognition approach (Scaife and Rogers, 1996)

• explain how we interact with external representations

• Identify the cognitive benefits and costs of using different representations

• provide principles to inform their design

Computational offloading

• the extent to which different external representations (5, five, |||||, V) reduce the amount of cognitive effort required to solve equivalent problems

45 x Forty-five times twelve IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XXXXV12 IIIIIIIIIIII XII

• the amount of cognitive work the learner is expected to do when using the external representation

“Dynalinking”

• linking of multiple representations so that making changes in one makes changes in the other (e.g., Pondworld)

• Dynalinking allows people to 'read' and connect information explicitly, helping them to understand the relationship between the two

Module Computatio

nal

offloading

Form of MM

interactivity

Problem

Solving

Activity

Learning

Process

1. PondWorld

Simulation

High Click and tell

animation

learning of

feeding

relationships

in ecosystem

Factual

knowledge:

feeding

relationships

2. IntroWeb Medium dynalinking

between

animation and

formalism

Make links

between

animation and

formalism

Canonical

forms in

foodweb &

mapping

between

abstractions

3. LinkWeb Medium dynalinking

between

animation and

formalism

complete

partially

constructed

formalism

Canonical

forms in

formalism &

mapping

between

abstractions

4. EraserWeb Low dynalinking

with facility for

annotating

diagram

use formalism

to reason

about

ecosystem

behaviour

Temporal

effects of

extinction of

species

Mixed realities

• What is Mixed Reality? - physical world is combined with virtual world

• Coupling the familiar with the unexpected

• Offers new opportunities for learning and understanding, e.g., reflection

Mixing colors

• Very familiar activity using wet paints and brushes

• What happens when you provide new ways of mixing colors?– e.g. combining digital paints to cause a physical

object to change color

• What do young children do?• How do they reflect upon the ‘old-new’

experiences?

The Chromarium project

(i) Wet paints

(ii) Combine digital colors

(iii) Combine physical cubes with colored sides to mix digital paints on a wall

(iv) Combine digital colors to cause a physical windmill’s sails to rotate and change color

(ii) Move digital colored discs on top of each other, resulting in new digital

colors

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

(iii) Place two physical cubes next to each other on a tabletop resulting in new digital color appearing on wall

QuickTime™ and aYUV420 codec decompressorare needed to see this picture.

(iv) Move digital windmill sails causing physical windmill sails to

move and change color

QuickTime™ and aYUV420 codec decompressorare needed to see this picture.

Findings from Chromarium study

• Children were most inventive and creative when mixing colors using the physical cubes

• Encouraged more collaboration and explaining of the ‘what’ and ‘why’

Summary: From icons to mixed realities

• Many cognitive benefits of using external representations in learning– understanding– reasoning – problem-solving– reflecting

• But can’t just assume a picture is worth a 1000 words or an animation a 1000 pictures or a mixed reality is better than a physical reality…

• Understand the properties of each and apply principles of design

Thanks

• All my colleagues and collaborators, especially those at the Interact Lab, Sussex University and those working on the Equator IRC in the UK