Heritage Visualisation: Lessons From The Fun Side

Erik Champion, Curtin University PISA 9 SEPTEMBER 2014

abstract

• heritage visualisation and serious game design

• major concepts and issues in the field

• learning from game design

• problems that arise when entertainment, heritage, history and education collide

Why Virtual Heritage?• Background in architecture, (art

history) and philosophy

• PhD with Lonely Planet in VEs for travel and tourism

• Taught interaction design and game design at UQ, UNSW, Massey

• Project Manager at DIGHUMLAB (Denmark)

• Professor of Cultural Visualisation, Curtin University, Perth, Western Australia

Cultural Visualisation?Feature Science Art Culture

Reusable data Yes No Seldom

Standard tools Yes Seldom Thematic & communal

Clear research question Yes Seldom Depends

Null hypothesis Yes No Not often

Extensible Mostly Seldom Important

Falsifiable Yes Seldom Difficult

Stored Surely Unlikely Vital

Visualisation is creativeProblem: A farmer was not allowed to build a barn to shelter his horses. Solution: He was allowed to build furniture.

Virtual Heritage is..• “[It is]…the use of computer-based interactive

technologies to record, preserve, or recreate artifacts, sites and actors of historic, artistic, religious, and cultural significance and to deliver the results openly to a global audience in such a way as to provide formative educational experiences through electronic manipulations of time and space.”

• Stone & Ojika, 2000

What Is Left Out?• Beliefs, rituals, other cultural behaviours and activities?

• Traces the level of certainty, and authenticity of reproduction and reveals process? I.e. scholastic rigor?

• Sensitive to the needs of audience & shareholder?

Virtual heritage is the attempt to convey not just the appearance but also the meaning and significance of cultural artefacts and the associated social agency that designed and used them, through the use of interactive and immersive digital media.

Visualisation & Simulation• “to form a mental image of something incapable of being

viewed or not at that moment visible… (Collins Dictionary)...a tool or method for interpreting image data fed into a computer and for generating images from complex multidimensional data sets” (McCormick et al. 1987).

• “The purpose of a simulation is to study the characteristics of a real-life or fictional system by manipulating variables that cannot be controlled in a real system…While a model aims to be true to the system it represents, a simulation can use a model to explore states that would not be possible in the original system.” (Beat Schwendimann, 2010).

For sites too fragile inaccessible damaged scattered or dangerous

VH is not VR• “Virtual reality is the use of computers and human-

computer interfaces to create the effect of a three- dimensional world containing interactive objects with a strong sense of three-dimensional presence.”

• The importance of using HMDs or CAVEs, for VR apparently requires “a head-tracked, usually a stereoscopic, display that presents the virtual world from the user’s current head position, including the visual cues required..”

New mediathe act of reshaping the user experience of exploring realms or worlds

through the innovative use of digital media. !

Wild Reindeer Exhibit Gagarin Interactive Iceland

INSTANTAR.ORGFraunhofer IDG CHESS project-Markerless tracking

KINECT-BOLOGNACINECA APA REUSABLE GAME

http://www.cineca.it/en

Definitions NEW MEDIA: the act of reshaping the user experience

through the innovative use of digital media. VIRTUAL HERITAGE: convey the appearance, meaning,

significance and social agency that designed and used cultural artefacts and sites, (through the use of interactive and immersive digital media).

NEW HERITAGE: re-examine the user experience that digital media can provide for the understanding and experiencing of tangible and intangible cultural heritage

! Erik Champion, in Y. E. Kalay, T. Kvan, & J. Affleck, New Heritage: new

media and cultural heritage. New York: Routledge, 2008.

Edu-retainment

• Game based learning, serious games, playful learning, game-based learning…

• “Anyone who thinks there is a difference between education and entertainment doesn’t know the first thing about either.”—Marshall McLuhan, Communications Theorist Sims: Virtual Montecello

Four Hypotheses• Social learning is inter-active but Culture is also

materially embedded or embodied.

• To teach and disseminate immersive Digital History and Virtual Heritage, interaction and the learning that results from that interaction is crucial (see Mosaker, 2001).

• To improve interaction, examine games and why they are so successful; academic literature suggests games are best examples of interactive digital engagement (references in Champion, 2008 et al.).

• Game-based interaction has to be modified for DH/VH.

4 Issues With Culture

1. Definition (relation to place and inhabitation)?

2. How is culture transmitted?

3. Transmit local situated cultural knowledge to “others”?

4. VH: how to transmit via digital & augmented media?

Central Point• Games are great learning environments

• Except for Cultural Significance, history and heritage

• Conclusion: problems and solutions

• Technology=barrier but not issue: learning is the problem.

• Which historical principles are used, learnt and applied?

• Inhabitants’ points of view (heritage) is missing

• Scholarly cycle incomplete, community cycle inextensible

Five Features of Place1. A place can have a distinct theme, atmosphere, and contextually

related artefacts.*

2. Some places have the capacity to overawe.

3. Place has the power to evoke memories and associations.

4. Place has the capability to act as either stage or framework in which communal and individual activity can ‘take place’.

5. Place has the ability to transmit the cultural intentions of individual participants and social ‘bodies’.

* Place is a process not a product, and can consist of multiple interpretations, conflicts, and a unique combination of borrowed histories. (Doreen Massey).

Why 3D?• To evoke +communicate historical situations or heritage values find

deeper understandings not simply memorize facts (Bloom, 1956).

• Place is a cultural setting, it gives cultural interaction a time and a location, Crang (1998, p.103), “Spaces become places as they become ‘time-thickened’”

• Places do not just organise space, they orient, identify, and animate the bodies, minds, and feelings of both inhabitants and visitors.

• Cultural presence: a feeling in a virtual environment that people with a different cultural perspective occupy or have occupied that virtual environment as a ‘place’.

Placeness Is More Than 3D

secondlife parodyyoutube (French)

Croquet

http://www.moddb.com/mods/sourceforts

Drawing interfaceshttp://www.drawastickman.com/episode1

Crayon Physicshttp://steamcommunity.com/app/26900

https://www.youtube.com/watch?v=L9YR_5SpLCg

Blender 3D-Jim Pachal

Why Recreate Place-ness?

• Fix locations in the memory (hippocampus).

• Reveal design based on scale and senses.

• Reveal limitations or principles of historical 2D images.

• Provide a heightened sense of difficulty, occasion, ritual, social proxemics (social hierarchies).

• Afford a sense of place: peripherality, centre directionality.

• Cultural landscapes affected by topography and climate and proximity to resources.

• Parts of language affected by geolocation of cultures.

iSphere copyright Paul Bourke

Monkey Brain, Human BrainCaption: The flow of object information in a monkey brain (left) and a human brain.

Credit: Sabine Kastner, Princeton University !

Culture is a feedback loop• A visitor perceives space as place, and inhabits (modifies

a place), place 'perpetuates culture’, and thus influences the inhabitants in turn.

• We might say that social behaviour is behaviour between two or more people.

• Cultural behaviour is a subset of social behaviour, where behaviour is governed by or understood in terms of a cultural setting.

• As culture almost inevitably involves transactions, there must be objects of shared transactional value.

Serious Games in CH• Gaius Day, Augmented Reality on site

• Roma Nova Petridis 2012

• Canada: A People’s History Game OR building Detroit

• On site historical (Schrier 2006)reliving the revolution

• Eduventure I: (Ferdinand 2005) Rhine valley history, played in real Marksburg castle, on tablet, webcam ARToolkitPlus

• Virtuso Arts history (Wagner 2007), sort a collection of artworks or monuments

• ThIATRO art history (Froschauer et al, VSMM 2012)

• Escaladieu (IRIT 2010) Abbey in Pyranees, 3D AR

• Studierstube ES game handheld AR platform

• Strategy eg Battle of Waterloo (BBC)

Video Games Pros & ConsFactors Weaknesses Strengths

Interaction Agency destroys historic causality. Simplistic interaction, may be difficult for older audiences. Helps teach interaction design.

Engagement Educational games: worst of both worlds? Well-known & popular.

Learning How to promote heritage & knowledge transfer. Learn by trial and error. Leveling allow for skills learnt

Technical issues Often contains many bugs. Often platform specific. Speed, lighting, avatar design, peripheries, networking

Support Support by the actual company can be slow, and they may avoid listing intended future features.

Community support (internet forums).

Game development

Non proprietary formats, changing game engine code requires high coding skills

Education discounts available, some games are easily “modded”.

Access/ cost Expensive software development kits and commercial licenses. Expensive.

Take them home, personalize modify and share them.

Institutional value Not taken seriously. Employability for students.

PALADIN-QUMULUSINTERACTIVE WEATHER

cryengine4

unreal engine-UDK

Kinect 1/2: voice + skeletonSkyrim has motion tracking and voice commands

Type to enter text

Type to enter text

LBP1+2

• http://www.mediamolecule.com/blog/article/kareems_talk_from_learning_without_frontiers_2011/

Games For History1. Play and and answer questions

2. Play and classroom discuss and debate authenticity

3. Role-play with games, puppets, or narrators

4. Mod cities, empires events based on theories

5. Film events etc. using Machinima tools

6. Combine images or panoramas with other media

7. Design past artefacts, events, rituals or customs

8. Create VEs using games and game mods or using VR

Playing History Plague – Slave trade - Vikings

Challenge: ..the belief that it is exciting to learn about history.

Integrates learning and playing in a way that engages pupils and gives them a concrete feel for the historical time and

setting Solution:

Platform: Mac/PC, single player, browser Technology: 3D Unity game engine

Playtime: Per game 60 minutes Target group: 9-14 years old

2. Discuss and debate

• Watch the movie, ‘Gladiator’ ..Identify an item of material culture (building, object, ‘thing’) that is important to the plot and structure of the movie, and..

• http://proteus.brown.edu/romanarchaeology08/4986

NB http://www.playthepast.org/

3. Role-playhttp://publicVR.org

OR video at http://vimeo.com/25901467

Assassins Creedflickr Niranjan cc shanewarne_60000

4. RTS-Mod cities empires

Kurt Squire:“We are interested in: the

processes by which players develop an interest in history,

what historical understandings develop, and if participation

has consequences for activities such as school.”

5. Film Events (Machinima)

http://www.sourcefilmmaker.com/

http://moviesandbox.com

http://www.thesims.com/de-de

Filmic 3D narrativesAPA reusable game: CINECA

6. Combine images, panos http://www.petermorse.com.au/vrar/vr/

Iphone: http://www.youtube.com/watch?v=s9sBtuCuju0 Technical description http://paulbourke.net/dome/UnityiDome/

Other pano examples http://paulbourke.net/transient/Beacon/beacontour.html

7. Design past artifacts, events, rituals or customsUnreal Tournament / Xibalba-Palenque

8a. CAVE via game enginehttp://cryve.id.tue.nl

8b. iVEs from VR systems

http://www.ntnu.no/ub/omubit/bibliotekene/gunnerus-1/mubil

8c.other mods

2001-2004 PhD thesis• Place versus Cyberspace: What creates a sensation of place (as a

cultural site) in a virtual environment in contradistinction to a sensation of a virtual environment as a collection of objects and spaces?

• Cultural Presence versus Social Presence and Presence: Which factors help immerse people spatially and thematically into a cultural learning experience?

• Realism versus Interpretation: Does an attempt to perfect fidelity to sources and to realism improve or hinder the cultural learning experience?

• Education versus Entertainment: Does an attempt to make the experience engaging improve or hinder the cultural learning experience?

Surround projection 2005

•

2005Unreal-music_Xavier Quijas Yxayotl-www.yxayotl.com.mp4

Film Students Into VEs

•Recreation  of  Maltese  temples,  the  students  created  ghost  narrators  by  videoing  themselves  recreating  imagined  rituals  and  inserting  these  translucent  videos  into  the  game  level.  This  was  done  in  less  than  12  weeks  part-­‐time  by  3  students  (undergraduate)  in  2005.  CAVI  (lower  left)  at  Aarhus  can  do  this  and  project  videos  onto  3D  statues  and  monuments.

Warping

3D and CMSUnity inside Moodle

Lazy Susan PanoramaHybrid 2D - 3D Navigation

Our$work$in$sensory$input/output$

meditation gameAndrew Dekker 2005

biofeedback zombiesAndrew Dekker

Lessons badly learnt• Very difficult to recreate original action scenes

and moments of discovery as game devices.

• Chinese players, familiar with a distorted version of the original, not aware their cultural knowledge was not accurate, did not appreciate being told this.

• Recreating linear narrative via game design is torturous.

• OR: simulate the procedural knowledge of rituals and symbol-making via thematically-akin interaction..

• ref: Game Mods: Design Theory and Criticism

Games aren’t Challenging?• A rule-based formal system with a variable and quantifiable

outcome, where different outcomes are assigned different values, the player exerts effort in order to influence the outcome, the player feels attached to the outcome, and the consequences of the activity are optional and negotiable. (Juul 2003, para 15).

• A system in which players engage in an artificial conflict, defined by rules, that results in a quantifiable outcome. (Salen and Zimmerman, 2004).

• A challenge that offers up the possibility of temporary or permanent tactical resolution without harmful outcomes to the real world situation of the participant (Champion, 2006).

Prescriptive or Procedural learning?

Gamer: Reach objectives as quickly and vividly as possible. Activity

Tourist: Enjoy highlights safely and conveniently. Viz: Weekend in Capri

Traveller: complete tasks via local affordances. Activity: Myst

Archaeologist: Discover past via examining material remains, geographical changes,

epigraphy etc.

Viz of process: ArcDig, detective

series?

Anthropologist: Understand the beliefs roles and relationships of inhabitants in context. geographical changes, epigraphy etc.

Hermeneutic: Myst, Sims? Oblivion?

Problem: Interaction /History• Ritual knowledge: Match artefacts with events to progress through time

• Memetic Cause &effect (Guess results or memes to progress history)

• Extrapolate from clues in NPC dialogue

• Role-play minor characters, “History” not affected

• Counterfactual histories (create many possible worlds)

• Augment virtual world with historical or current media

• Sentiment analysis (observe the emotional impact of events on NPCs)

• Separate lies from truth to progress

• Mimic NPCs (as a kind of reverse Cultural Turing Test)

Problem: Avatars

• Realistic depiction

• Social behaviour

• Interface issues

http://www.interactivestory.net/

Eric Fassbender: Macquarie Lighthouse

Problem: Inhabitants’ PoV• Can users learn via interaction the meanings and values

of others, do we need to interact as the original inhabitants did?

• How can we find out how they interacted?

• Can the limited and constraining nature of current technology help interaction become more meaningful, educational and enjoyable (Handron & Jacobson, 2010)?

• How do we even know when meaningful learning is reached?

Problem: Rituals• Attention/focus

• Social judgement

• Territoriality

• Social Proxemics

• Being “in the flow”

• Physical delineation (profane vs sacred)

• Event-based or regular

http://www.youtube.com/watch?v=MqMXIRwQniA Image: http://www.virtualtripping.com/google-earths-rome-reborn/ 2008

Problem: Sensory immersion

Affective Process

Kinect 1/2: voice + skeletonSkyrim has motion tracking and voice commands

Problem: Integrate Text+Modelhttp://gap.alexandriaarchive.org/gapvis/index.html#index

Problem: Violence• No realistic humans

• No social judgement

• No time to think

• Gun based genres are commonplace

• Weaponry skill can be easily levelled up

• Typical single player

• Demographics

Alternatives To Violence• Reflexivity: A reflective space,

where players relax & consider the consequences of their actions

• Performativity: Players asked to perform or orate and present their experience of the VE in class.

• RPG Virtue Ethics: Characters change in relation to development of virtue ethics.

• Consequentialism: Consequences of player actions affect their future gameplay. through the game.

• Creative Uses For Weapons.

• NPC distaste and disparagement: they discourage violence.

• Biofeedback: Performance based on calmness.

• Expressive and embodied modes of interaction.

• Non-violent competition.

• P l a y e r s b e c o m e m o r a l l y accountable for their actions.

• Ritual or mythical use of weapons.

Evaluating VES - People• Task performance (quantitative or

qualitative)

• Likert or statistical evaluation

• Extrapolated understanding

• Personal ‘sense’ of cultural presence

• What do they choose next (exit strategies)

• ‘Teach the teacher’ et al methods

• Excitement recorded from biofeedback

http://www.onzeonze.com.br/blog360/toursaofrancisco/index.html

http://www.onzeonze.com.br/blog360/toursaofrancisco/index.html

UNESCO1. Create list and host online available 3D heritage

models, concentrate on Australia and the Pacific region.

2. Collate and archive VH resources via HuNI & AARNET.

3. Show community groups how to develop and modify their own online virtual heritage models and sites.

4. Provide training material on 3D capture and design that can be expanded by others. Test via workshops at Curtin & partners.

5. Recommend long-term archive guidelines, ways of linking 3D models to scholarly publications & resources.

6. Advice & modify suitable FOSS-Free and Open software and creative copyright for 3D heritage.

The HIVE@CURTIN

Tiled Display

Jeffrey Jacobsontalking at Curtin

Jeffrey JacobsonCurved Screen, HIVE, Curtin

The cost of Stereo VR

Linked Open Data And Motion Capture

Conclusion• Games as Virtual Environments may connect more people,

more thematically without competing with book learning.

• Background research needed for public vs. scholar needs.

• Game conventions ‘work’ but meaningful learning elusive.

• We lack interactive and immersive digital history projects that are meaningful and engaging learning experiences.

• Mixed reality in history and heritage has many advantages but few working exemplars.