1  

Electronic Art: The Synaesthetic Effects

Kelly Teh, University of Westminster

Available at: http://www.harpylistens.com/TheSynaestheticFX/

Introduction

I have been travelling back and forth between art and technology for several years. However, I

discover something new in every journey I made. In fact, my last job as a technologist gave me the insight

into the relationship between technology and creativity.

While art has been my current location, I intend to explore the relationships between technology

and art based on aesthetic perceptions and philosophical conceptualisations. However, this essay will

focus on research in audio-visual art by means of pervasive technology. The combination of auditory,

visual or more senses in the form of an involuntary experience is also known as synaesthesia. The

discovery of this subject since then has aroused the interest of many scientists and artists. Francis Galton

who was the first to document the study on this neurological based phenomenon was only referred to two

decades later as scientists learned more about the brain (A.Jensen 2007). By the late 19th century, many

visual artists, notably Wassily Kandinsky, who claimed to have synaesthesia, produced abstract paintings

or also known as ‘colour music’. However, since the emergence of modern technology began, some artists

started to explore sound in relation to visual art by simulating synaesthesia in their work. For example,

the Russian composer from the 19th century, Alexander Scriabin was interested in unifying his music with

colour and made several attempts with colour projections in his music performance. Due to the

technology that was less advanced during then, Scriabin was never satisfied with the optical effect.

Nevertheless, art never stopped to evolve along with technological progress. The acceleration of

the technology progress has also created a shift in the conceptualisation of synaesthesia in art, especially

electronic art. Synaesthesia was no longer seen only as a neurological based condition experienced by

certain artists but as a means of working in multi-sensory mode. Therefore, my research defines

synaesthetic effects as the process of unifying sound and visual by means of technology rather than a

speculation on the actual neurological based phenomenon. It is also a transmission of data between

human and machines, which is described as tele-synaesthesia while H.Heyman (1995) states:

‘The quintessential difference between natural synaesthesia and tele-synaesthesia lies with the fact that

electronic media are extensions of ourselves, allowing for synchronisation and corrective interventions,

thus bringing about a change/modification of our sensorial system and conceptions. In this respect, a

processual, interactive approach to reality can be directive, since meaning is not a static data, but rather

a type of activity.’

The concept of synaesthetic effects in electronic art has also created evident dialogs between

visual art and sound art. Human most commonly used sense in general is sight, which allows us to obtain

most of the information around us. In reference to that, I am interested to use my research to analyse the

context of sound in the contemporary art world and the impact that it creates on human communication

and interaction. It is also known that these synaesthetic effects produced by technical innovations

constantly create a whole new human experience. Therefore, the question of how it has influenced human

mood, behaviour and cognition is also being raised.

Art: Technology

  2  

Wilson (2001, pp.13) states that, ‘many histories of technology are essentially histories of

invention’. Engineers and technologists are constantly creating new tools that extend the possibilities and

functions of human physics. The cultural change in society has, of course also influenced the way

technology has evolved and vice versa. The first part of my research explores how audio-visual technology

evolved and the impact it has on art practices that are related to the fusion of sound and image.

According to Michael Rush, technology-based art was initiated by inventions outside the world of

art and it has redirected art in the 20th century into the areas that were once dominated by engineers and

technicians’. (M. Rush, 2001, p.2). The history of technology often illustrates how inventors, scientists,

engineers and technicians made their contributions, which have greatly influenced the social and cultural

traditions. While history of art tends to consider the technological evolution as part of its definitions, the

following paragraph resulted in my research can also be considered as an argument against Rush’s

statement.

British photographer Eadweard Muybridge invented the Zoopraxiscope in1879, a device for

projecting motion picture, as he was interested in capturing the movements of human beings and

animals. The fact that Muybridge’s art-based invention had inspired other inventors, such as Thomas

Edison, motion picture technology was developed. Edison the most notable American inventor who

invented tools that contributed to mass communications were also considered as media inventions. His

contributions to the media technology such as the phonograph, radio, kinetoscope and vitascope, sparked

the growth of sound and motion picture production. Indeed, these developments gave rise to the

traditions of experimental music and film with the use of sound and images. In fact, live music

performance and sound recordings later accompanied silent film, which greatly impact the audience

perception with that new experience. Given electricity and light as a medium for artists to experiment

with, kinetic art became an increasingly popular type of work. The most common examples of kinetic art

are in the form of sculptures such as Laszlo Moholy-Nagy’s Light Space Modulator (1922-30); in which

electric motors are used to rotate the steel sculpture at a high speed to create light reflection from the

electrical illumination (Shanken 2009). However, British inventor Alexander Wallace Rimington presented a

remarkable invention called the Colour Organ in the late 1800’s, which extends his influence to the work

of many artists. Rimington was interested in the correspondence of sound and light vibrations and had

the idea to use the machine to visualise the idea behind music through light projection of colours. His

invention initiated several visual performances such as Scriabin's synaesthetic symphony Prometheus: A

Poem of Fire. Despite source of light being a common medium for artists to experiment with, such

practices are not considered as contemporary art during then, as opposed to the traditional visual art that

is static.

Studies in Zoopraxography arranged for the Zoopraxiscope by Edweard Muybridge (1893) Mule Bucking and Kicking

  3  

[Electronic print] Available at: < http://www.geh.org/fm/precin/htmlsrc4/m200113080001_ful.html#topofimage> [Accessed 3 April 2011]

Alexander Wallace Rimingtons colour organ (1910) [Electronic image] Available at :< http://www.lumen.nu/rekveld/wp/?page_id=340> [Accessed 3 April 2011]

Since the late 20th century, however, technology-based art became increasingly incorporated into

contemporary art practices especially with the advent of digital technology. The rise of the digital culture

was indeed a new revolutionary phenomenon as it continued to impact human life and transform culture

extensively. Society relies heavily on technology such as mobile phones and the Internet and this drives

the artists to explore within this new realm in response to questions raised about how it has reshaped

their existence. This inevitably transforms traditional art into new forms such as multimedia art,

installation art, interactive art and virtual reality. Artists are also relying on the fact that digital technology

offers more possibilities in creating cross-sensory work, which can be described as Gesamtkunstwerk, the

term used by Richard Wagner as he describes his idea of unifying works of all art. Digitisation of art,

which offers a cost effective, yet wider and faster distribution, is becoming a preferred method. It has

changed the perception of traditional art such as painting and photography. The Internet gives the

audience the opportunity to view and appreciate art within the virtual world as opposed to being

physically present in a white cube gallery. In other words, the decentralization of art began with the

emergence of Internet because the mediated communication eliminates the geographical distance.

Consequently, artists are able to distribute their art across different cultural regions within the virtual

world, which helps to transcend the cultural differences. However, Shanken states that artists also started

using computer networking as a bona-fide medium, in which Roy Ascott called it as telematic art (Shanken

2009). The potentials of collaboration and interactivity that Internet has, attracts the attention of many

artists to explore the medium. In fact, there has been a rampant movement in Internet art among

professional artists as well as amateurs due to the availability of this affordable technology, which has

become integrated into human daily life.

Synaesthetic art: Human Communication and the experience

Shanken (2009) suggests that paralleling the intrinsic temporality of music and cinema, artists

increasingly have set art in motion in such a way that the work can only properly be perceived as

durational experience. Shanken’s suggestion leads to further discussions pertaining to the effects of

sound and visual representations on human communication and experience in this chapter.

Because the sense of sight and sense of hearing are the most important senses, audiovisual

integration can be a form of enhancement on human spatial and temporal perception, which enables both

senses to correspond with each other. By presenting art in the spatial-temporal context using sound and

visual representations, artists are able to communicate a certain expression of emotions and messages.

  4  

For this reason, artists in the search of various aspects of human experience closely associate art and

synaesthesia. The most common type of synaesthesia is the association between colour and sound

because human beings usually experience emotional reactions to colours and pitch. Indeed, many

synaesthesia born artists such as Kandinsky and Klee tried to manifest their experience in their paintings

but that did not enable everyone to fully experience it as such because paintings are merely visual

representations. However, the exploration on psychological effects of intersecting audio and visual

sensory information led artists into simulating the experience of synaesthesia by means of technology.

For example, the German movement in the early 20th century, which explores the relationship of cinema to

music, painting and light sculpture, creates new art forms that encompass such inspirations. Walther

Ruttmann who wrote about ‘painting with time’ presented his first public event with his abstract film

Lichtspiel Opus 1 with a live string quartet (Keefer 2005-2009). In art history, Ruttmann is believed to

have enriched the language of films with this new form of film expression or new form techniques.

Consequently, such influence can be seen in later born artists such as Oskar Fischinger, who writes that,

"Of this Art everything is new and yet ancient in its laws and forms. Plastic - Dance - Painting - Music become one. The Master of the new Art forms poetical work in four dimensions…Cinema was its beginning…Raumlichtmusik will be its completion." (Cited in Keefer 2005-2009)

PastFinderLtd, 2008.Lichtspiel, 1921 - www.pastfinder.de.[Video Still] Available at:<http://www.youtube.com/watch?v=9V8DjVBqHBA>[Accessed 3 April 2011]

In fact, Forde (2005) suggests that the creation of electronic art that involves technical requirements

inspires artists to reflect on the consciousness of highly un-technological experiences-physicality, human

cognition and perception. In addition to that, Forde (2005) also underlines another important statement

about the merging of all art forms, which contain visual and auditory information that became a gesture

in social political discourse. This is clearly reflected by Wagner (1927) who asserts that,

“ The great united art-work, which must embrace all the genres of art…we cannot recognise as the arbitrary need of the individual, but only as the inevitable associated work of the humanity of the future”(cited in Forde 2005, p.12)

Wagner’s statement certainly corresponds with the belief and principles in my own art practices. For this

reason, my research argues for a greater insight into how multi-sensory art impacts human

communication and experience based on the aesthetic and cognitive aspects. Although electronic art may

seem far from human likeness, artists always discover new ways to express and to speak through their

work because symbolism in art can be referred to the origin of verbal and nonverbal language. According

to Robert Logan (2007), a media ecologist who did research in evolution of language, the attempt to link

verbal language and art is because of the realisation that the visual arts and music are both percept-

based. In a multimedia performance piece called ‘In Seven Days’, the artists Tal Rosner and Thomas Ades

tell the Genesis story of creation with live orchestra music and abstract images of nature. The way the

story has been told may be explained as Logan (2007) further describes the possible link between verbal

language and art is due to the physicality of the artistic medium, conceptual, symbolic and

representational nature of the arts.

  5  

harpylistens, 2011. In Seven Days by Tal Rosner and Thomas Ades. [Video Still] Available at: <http://www.youtube.com/watch?v=yFllUjGKGOQ>[Accessed 3 April 2011]

The growing acceptance of this type of art that was referred to as Avant Garde art enables many

artists to employ these ideas in the increasingly popular art genres such as visual music and abstract film.

This can be traced in the mid 1960’s when the experience of synaesthetic effect soon reached the rock

music concerts when artists began to collaborate with bands in the mid 1960’s such as The Soft Machine,

Pink Floyd and Jimi Hendrix. In Liquid Light Show created by Mark Boyle, an artist who pioneered visual

music in the UK music industry, sources from liquid and light were used to depict the live music in

emotional visual terms. However, the combination of the psychedelic sound of rock music and visuals that

animate the sound creates a trance experience for both performers and the audience. This is why it can

also have a positive or negative impact on people’s behaviour.

therealgaryhill, 2009.Psychedelic Light Show UFO 1967: Soft Machine. [Video Still] Available at :<http://www.youtube.com/watch?v=QWV2Co4X4pw>[Accessed 3 April 2011]

Due to my own curiosity in the effects of music visualisation, an experiment was carried out in

which I composed a piece of minimal music with presumptions on the association between the sound and

visuals made. One of my peers then made visual representations of the sounds in the music with her

painting. The result shows that our colour association with the sound corresponds more with each other

as opposed to the texture and shapes that were rather different. Does that mean that I did not

communicate my sense of visualisation well through the music? Although art aims to communicate in a

way that simulates certain thoughts and emotions according to the artist’s intention, nonetheless, human

perception is no more than a subjective phenomenon. In the discussion, Greated (2009) also mentions

that artists do not always have control over the experience due to the dominance of the viewer’s role

within the work and the impact of situational activation of space. Some people may perceive art as a

decorative element despite the intended concepts embedded by the artist. Similarly, some people may

choose to filter out the sound while focusing on the visuals or vice versa as one of those comply with his

or her perception better. The widespread release of 3D films that use a huge amount of auditory and

visual information to simulate reality for the viewers may not be applicable to everyone as it may be an

overload to the brain for certain people.

  6  

Minakshi Singh, 2011, Dust Covered Letter

Despite the inevitable challenges, art creation, technology and understanding in human

experience continue to evolve in parallel to communication. As mentioned in the previous chapter, digital

technology enables artists to generate an interactive experience for the audience, which elevates the level

of communication between the artists, the audience and the art medium. In Synaesthesia Urbania, a

collaborative project initiated by Toy Satellite and a Korean community, public participations in several

countries are included through the mobile phone Multimedia Messaging System (MMS). A combination of

text, images and sound sent by the public are used in a live audiovisual performance, which communicate

the diverse cultural references in each country. This creates an enhancing experience for the audience

because they are not only engaged in the visuals and sound but are also able to facilitate the events that

involve cross-cultural communication.

Sound: The Changing Role In Art

When Greated (2009) wrote that ‘The general acceptance of vision as the dominant sense in

arts, philosophy and science has recently been questioned and challenged’, she suggested that there is a

growing interest in the role that sound plays in the ideas of art perception. Sound art and music, which

are overlapping genres inspire artists to use sound to manipulate visual and transform the human

experience instead of mere accompaniment. This can be traced since the emergence of an electronic

environment that initially created a bridge between sound art and architecture (Shanken 2009). In addition

to that visual aesthetics of architecture, the acoustics also contribute extensively to the listening

experience, which is why the association between sound and space or location is widely considered by

artists these days.

In fact, Susan Phillipsz, who was the first sound artist to win the Turner Prize in 2010, merely

played recordings of herself singing folk songs via loudspeakers installed beneath bridges in Glasgow.

Despite the simplistic idea, Phillipsz’s work changed the way people see the bridge and the acoustics

under the bridge enhances the listening experience. The Turner Prize result that year has indeed raised

the question about the role of sound in the contemporary art world. Nevertheless, sound based work can

also get very complex as artists collaborate with engineers to further investigate the characteristics that

can be employed within the artwork. In 1970, E.A.T (Experiments in Art and Technology) created The

Pepsi Pavilion for the 70 Expo in Osaka, Japan, which was an outstanding experiment in combining sound,

visuals, electronics and architecture. Laser generated patterns responding to sound are projected into the

dome, which generates a three dimensional image (Shanken 2009). The sound system is also set up in a

way that optimises the sound motion and the acoustics inside the dome. Another piece of work in the

sound and space context created by David Byrne is called Playing The Building. It is a sound installation

  7  

that took place in the Battery Maritime Building in New York City. Devices that produce vibration, wind or

striking, were attached to the building structure causes the building elements to produce sound

themselves as they resonate and oscillate so that the building becomes a giant musical instrument itself.

Pepsi Pavilion, Expo '70, Osaka, Japan. The Pavilion << Into the 21st Century >>[Video Still] Available a t: <http://www.zakros.com/projects/pavilion/overview_new.html>[Accessed 3 April 2011]

Playing the Building, Battery Maritime Building, New York, NY, 2008 [Electronic Image] Available at: <http://www.davidbyrne.com/art/art_projects/playing_the_building/#>[Accessed 4 April 2011]

In addition to that, soundscape, which is a subject of acoustic ecology inspires artists to use

environmental sound as source material for their artwork. Because location sound recording intends to

capture the experience the natural acoustics of the environment, the playback of it can be considered as a

sound visualisation of the place. David Leddy who wrote a theatrical piece called Untitled Love Story

(2011) will only be using electro-acoustic soundscape in the performance, which will only be taking place

later this year. Therefore, the audience will be ‘watching’ the performing with their ears only. I am

convinced that by redirecting all attention to sound, there will be a greater emotional impact that will

expand the audience imagination in visualising.

Sound is also important to human communication, as the voice in verbal language is the primary

form of expression. Logan (2007) comments on the brain that consists of one part, which is associated

with verbal language and another part, which is associated with visual and auditory perception are both

engaged in an artistic activity. Therefore, voice also makes a powerful medium that can be incorporated

into art. In Messa di Voce (2003), Golan Levin uses voice or speech to stimulate abstract visuals, which are

highly expressive in order to correspond with the voice. According to Levin (2003), the project explores

abstract communication and synaesthetic relationship, whereby the audience can experiment with their

‘extended vocal techniques’ in speech that correspond with the graphics. Levin later created another piece

called Ursonography (2005) and it projects the relationship between speech and text in a poem. A

performer recited a poem in which the speech was reduced to its most abstract musical elements or

phonemes. The sound of the vocal was translated into text according to the rhythm and intonation of the

sound. This is nothing similar to our daily text based communication whereby we often find it difficult to

perceive the rhythm and intonation, which depict the emotions in the message. In fact, Goose (1998)

  8  

argues that ‘better recall can be attained if information is received aurally as opposed to being read, and

humans can react faster to auditory stimuli than to visual stimuli’ (p.122). According to Goose’s

argument, sound is capable of carrying more emotional values, which has a more immediate impact on

human beings.

Messa di Voce Installation ( ArtRock 2005), 2005 [Electronic Image] Ursonography, 2005 [Electronic Image]

Available at :<  http://www.flong.com/projects/messa_inst/> Available at :<  http://www.flong.com/projects/ursonography/>

[Accessed 4 April 2011] [Accessed 4 April 2011]

Conclusion

My role as a practicing artist, engineer and a Buddhist practitioner has inspired me to apply the

functionality of technology in the conceptualisation of my art practice, which do not only intend to create

an impact on the audience sensory experience and perception but also on myself. My research has given

me the insight into the relationships of various aspects of the art creation and technology and science,

which continue to inspire and to give shape to the cultural change.

Synaesthesia remains as a highly interesting research field where both scientists and artists

continue to conduct experiments to better understand the effect it has on human beings. In fact, UK

Synaesthesia Association recently held its annual conference that was attended by researchers from

various faculties such as computer science, language science, psychology, musicology and multimedia art.

The papers presented by the participants clearly reflect the strong connections between audiovisual art

and human perceptual experience and interaction, which fall under the study of psychology. Wilson (2001)

underlines the relationship of technology and science and the role of artists by commenting on Cyril

Stanley Smith’s book From Art to Science: Seventy-Two Objects Illustrating the Nature of Discovery as he

writes that,

“ In it, he observes that in the areas of chemistry, physics, and materials sciences, artists and artisans discover and use “subtle properties of matter” before they are even noticed by research scientists.”

Despite the expanding inclusiveness of science and technology in the world of art, the work of

art that intervenes in the aspects humanities is able to create new theoretical levels for understanding in

science and influence on cultural change. Electronic artists are inspired by technology to seek potentials

in unexplored territories where all boundaries between art and humanities can be transcended by creating

the notion of multi-sensory experience. Consequently, this increases our awareness of their interrelated

senses, which greatly influence our perception, behaviour and interaction. In addition to that, Francis

Bacon also suggests that ‘the understanding of nature often becomes clear only when trying to

manipulate it technologically.’ (Shanken, pp. 14).

  9  

References

1. Wilson S., 2001, Information Arts: Intersections of Art, Science, and Technology, The MIT Press, Cambridge, Massachusetts, London, England

2. Rush M., 1999, New Media in Late 20th-century Art, Thames and Hudson, London

3. Forde K. 2005, What Sound Does A Color Make?, Independent Curators International, New York, pp. 11-29

4. Shanken E.A, 2009, Art and Electronic Media, Phaidon Press, London

5. Cindy Keefer, 2005, "Space Light Art" - Early Abstract Cinema and Multimedia, 1900-1959 [online] White Noise exhibition catalog, ACMI Melbourne. Available at: <http://www.centerforvisualmusic.org/CKSLAexc.htm>[Accessed 20 March 2011]

6. Dr. Hugo Heyman, 1995, Tele-Synaesthesia: the Telematic Future of the Senses [online] Available at: <http://www.doctorhugo.org/synaesthesia/e-tsyn.htm> [Accessed 10 March 2011]

7. Toy Satellite, 2005, Synaesthesia Urbania [online] Toy Satellite. Available at: <http://www.toysatellite.org/syn-urbania/about/> [Accessed 10 March 2011]

8. Amber Jensen, 2007 ‘Synesthesia’. Lethbridge Undergraduate Research Journal [online] Volume 2 Number 1. Available at: <http://www.lurj.org/article.php/vol2n1/synesthesia.xml> [Accessed 10 March 2011]

9. Ward J., Moore S., Thompson-Lake D., Salid S., Beck B, 2008, The aesthetic appeal of auditory-visual synaesthetic perceptions in people without synaesthesia. Perception [online] Volume 37, pp. 1285-1296. Available at: <  http://www.perceptionweb.com/abstract.cgi?id=p5815> [Accessed 1 March 2011]

10. Greta Berman, 1999 ‘Synaesthesia and the Arts’. Leonardo [online] Volume 32 Number 1, pp. 15-22. Available at: <http://www.jstor.org/pss/1576621> [Accessed 20 March 2011]

11. Derek Yau and Rachel McCrindle, 2007 ‘MusicCam: an instrument to demonstrate chromaphonic synaesthesia’. Digital Creativity [online] Volume 18 Number 2, pp. 121-127. Available at: <http://www.informaworld.com/smpp/content~content=a778861403~db=all> [Accessed at: 7 March 2011]

12. Marianne Greated, 2009 ‘The nature of sound and vision in relation to colour’. Optic and Laser Technology [online] Number 43, pp. 337-347. Available at: <http://www.elsevier.com/locate/optlastec> [Accessed 7 March 2011]

13. Logan R.K., 2007, The Emergence of Artistic Expression and Secondary Perception [online] Available at: <http://www.physics.utoronto.ca> [Accessed 10 March 2011]

14. University of Wesminster, 2010, Referencing Your Work [online] Available at: <http://www.home.wmin.ac.uk/library/documents/Referencing-your-work.pdf> [Accessed 20 March 2011]

  10