a social semiotic approach to multimedia: the work of ......theory of multimodal social semiotics...

Running Head: A SOCIAL SEMIOTIC APPROACH TO MULTIMEDIA 1

A Social Semiotic Approach to Multimedia:

The Work of Gunther Kress as a Foundation for Design and Media Studies

Begüm Saçak, [email protected]

Erikson Institute

A SOCIAL SEMIOTIC APPROACH TO MULTIMEDIA 2

Abstract

Former linguist and semiotician Gunther Kress created the multimodal social semiotic theory,

which explains meaning-making process through non-linguistic modes other than written and

spoken language. In this paper, the foundations of the multimodal theory of communication and

its implications for the field of design will be discussed.

Keywords: multimodality, mode, social semiotics, design, linguistics, semiotics


Without integrative disciplines of understanding, communication, and action, there is little hope

of sensibly extending knowledge beyond the library or laboratory in order to serve the purpose of enriching human life. (Buchanan, 1992, p. 6)

A Social Semiotic Approach to Multimedia: The Work of Gunther Kress as a Foundation for

Design and Media Studies

Design thinking includes a form of communication through which designers or design

teams develop the best means to convey intended messages to an audience. Along with creative

thinking and clear objectives, designers, be it for instruction or for other purposes, need to grasp

the essentials of design.

…we might say that design thinking as the activity of thinking about design allows us to reflect on questions of who can design, and what can be designed. In this reflective mode, design thinking offers us ways to study the perceptions, expectations, and capabilities assigned to and associated with the theories and practices of designing (Cooper, Junginger, & Lockwood, 2009). In design terminology, it is possible to encounter the term design language. The word

language might appear to be a simple metaphor to express the design elements or processes;

however, it actually refers to a very sophisticated method of visual communication that warrants

attention to detail and an understanding of how the elements of design are used to speak to an

audience (Baldwin, 2016). If we dive into the interdisciplinary realm of human knowledge,

however, it will be possible to see how the study of language, linguistics, and the study of signs,

semiotics, paved the way for the language of design. The works of two scholars, Gunther Kress

and Theo van Leeuwen, created the theory of multimedia rooted deeply in the fields of

linguistics and semiotics. In this paper, the multimedia theory informed by linguistics and

semiotics will be discussed based on the work of Kress and his theory of multimodality.

Implications for how theory of multimodality can inform learning design will be given.


Background

The field of linguistics was still blooming when Chomsky (1965) proposed his idea of

universalism – a theory of language which will later gain worldwide recognition. Universalism is

the idea that every human being is born with a Language Acquisition Device (LAD) in their

mind, which help them acquire language at a very young age. The theory received some criticism

from some scholars such as Piaget (Piatelli-Palmarini, 1980). One of the main criticisms to the

universalism was that the theory accounts little for the evolutionary process and how this device

came to existence. Similarly, linguist and semiotician Gunther Kress criticized the theory

asserting that its asocial nature left nothing to society and human creativity (Hodge, 2013; Kress,

1998). In Kress’s mind, language was much more than Chomsky’s universalism – it should be

less linear and reductive, and more open to social and other dimensions (Hodge, 2013). In 1990s,

Kress and Theo van Leeuwen started talking about multimodality. In a seminal article, Kress and

van Leeuwen (1998) began to explain multimodal texts – texts which include a complex mixture

of words and images. An interesting gap arising out of the work of Kress and his colleagues was

their realization of multimodal texts and how meaning cannot be reduced to language only. In an

attempt to address this gap, based on Michael Halliday’s (1978) functional model of language, a

theory called Multimodal Social Semiotic Theory was formed by extending the theory of

meaning beyond language (Kress & van Leeuwen, 1996). In other words, the meaning making

capacity of various modes (functions) such as music, sound, action, and visual communication

are as crucial as the language itself.

Multimodal Social Semiotic Theory, Linguistics, and Semiotics


It is possible to trace back the origins of multimodal social semiotics to Halliday’s

fundamental theory of grammar. Halliday’s theory of language and meaning diverged

considerably from Chomsky as the emphasis was on text, which is defined as a process of

meaning-making in context (Halliday & Matthiessen, 2013) whereas Chomsky’s theory of

language is more concerned with innate linguistic competence. Halliday (1978) emphasized the

generative power of ‘the social’ which addresses “the close fit between social organization,

human action in social environments, and the meaning potential of linguistic forms and

processes” (Kress, 2010, p. 60). Kress and later his colleagues used Halliday’s theory of

grammar as “a means to ground a social critique in a close attention to the grammar of language”

(Iedema, 2003, p. 30).

Halliday (1978) developed systemic functional grammar – a tri-dimensional theory of

meaning (Iedema, 2003). Unlike some other theoretical explanations of language, systemic

functional grammar is about the specific functions; namely, the purposes of language use.

Halliday’s systemic functional grammar is about how speakers generate and communicate

meanings through “the generalized metafunctions that relate language to the outside world where

interactants and their social roles matter” (Haratyan, 2011, p. 260). The three main metafunctions

are ideational, interpersonal, and textual aspects of meaning. Ideational function comes from

human beings’ desire to interpret experience in terms of what’s going on around and inside us;

interpersonal function refers to interaction with the social world by negotiating social rules and

attitudes or how language is used to position and relate to people; textual function refers to

grammatical systems that create text that coheres within itself and with the context of situation

(theme) (Halliday, 2003; Nordquist, 2018; Muntigl & Ventola, 2010).


Halliday’s work is important because it enabled the analysis of “language from a social

semiotic point of view” by addressing the structures “above the sentence level” (Iedema, 2003).

Kress’s multimodal social semiotic theory derives from Halliday’s theory of language by

asserting the idea that “language is social” (Jewitt, 2006, p. 3), and linguistic account of meaning

is partial. Kress (2010) criticizes the European intellectual traditions as they put a great emphasis

on language in describing communication practices. Linguistic meaning has always been

regarded as the guarantor of rationality and knowledge (Kress, 2010). However, an important

question remains: Is language the most comprehensive means for communication?

Language constitute an important part of human communication but reducing communication to

language is not enough to explain complex interaction patterns human beings use to

communicate with each other. Communication is based on meaning-making process in which

different signs are used and interpreted in social interactions. Theory of multimodal social

semiotics was born in an attempt to explain meaning-making practices through social actions and

interactions (Kress, 1997, 2003; Kress and van Leeuwen, 1996, 2006). Though Hallidayan

approach to language also puts great emphasis on the social functions of language units and use,

social semiotics is based on the idea that “representation and communication always draw on a

multiplicity of modes, all of which contribute to meaning” (Bezemer & Jewitt, 2009, p. 183). In

other words, multimodal communication has more expressive affordances than language. Instead

of linguistic units, a multimodal communication can include but not limited to “image, writing,

layout, music, gesture, speech, moving image, soundtrack, and 3D objects” (Kress, 2010, p. 80).

In other words, rather than relying on abstraction, visual grammar is defined by “materiality”

(Kress, 2014).


Social semiotics also draws upon the field of semiotics - the study of signs. The works of

two semioticians, Swiss linguist Ferdinand de Saussure and the American philosopher and

logician Charles Sanders Peirce, are considered as fundamental in the field of semiotics

(Manghani, Piper, & Simons, 2006; Kress, 2010). Peirce’s idea of semiotics rests upon the

interaction between representamen (the form a sign takes), interpretant (the sense/meaning made

of a sign), and the object to which a sign refers (Manghani et al., 2006). “Readers” of signs are

agentive and transformative as they process the interpretants (meanings), which marks a point of

departure for news signs to be made (Kress, 2010). This semiotic process of meaning-making in

Peircian account depends primarily on readers or recipients of signs (Eco, 1984, Kress, 2010).

Figure 1. The Peircan model of sign (Nöth & Jungk, 2015, p. 666 originally from Introduction to Peircean Visual Semiotic (p.6), by Jappy, 2013, New York & London: Bloomsbury.


In contrast, Saussure focuses on linguistic sign which doesn’t necessarily correspond to

an object or referent, but there is an arbitrary relationship between the signifier (sound-image or

sound-shape) and signified (the concept that is signified).

Figure 2. Visual illustration of Saussure’s example

Saussure used the French word arbre to illustrate his point. A real tree in the world

(concept) is represented by a sound-image, and the nature of this relationship is arbitrary.

Different sound-shapes in different languages (such as the word tree in English or ağaç in

Turkish) represent the same signified (referring to a tree in real life) but signifiers are different

(i.e. different sound-shapes in other languages).

These profound ideas in semiotics influenced the study of meaning. Saussure’s

pioneering work marked a linguistic turn, in other words, social and cultural life has been

critically examined through texts or “the idea of textuality” since then whereas the “postsemiotic

discovery of picture” marked the beginnings of visual semiotics (Manghani et al., 2006).


The Art of Shaping the World through Social Semiotics

Kress and van Leeuwen (2001)’s theory of multimodal social semiotics was born as a

result of the increasing reliance of visual modalities rather than linguistic ones in our daily

communication practices. The creation of World Wide Web led to profound changes in

communication and our understanding of meaning-making (Vandendorpe, 2009). The distinction

between the visual and linguistic modes of communication grew to a point that a need to pay

attention to multimodality of signs and their meaning-making affordances emerged (Manghani et

al., 2006).

In multimodal social semiotics, signs are not arbitrary but rather, they are motivated; they

reflect the sign-maker’s interest in meaning making (Kress, 2003, 2010). A sign-maker has the

power to create and use signs as necessary.

…in Saussurian semiotics, if I want to be understood, I do so by learning the social rules of use of the semiotic resources… [i.e. learning a language]. If I don’t know them, I am in trouble. In Social Semiotics, if I want to be understood, by preference I use the resources that those around me know and use to make the signs which I need to make. If I am not familiar with those resources, I make signs in which the form strongly suggests the meaning I want to communicate [note that this is the opposite of arbitrariness’]. (Kress, 2010, p. 64) To illustrate multimodal social semiotic theory, a picture of a toy car was included, which

originally appears in Donald. A. Norman (2013)’s book, The Design of Everyday Things.


Figure 3. Lego motorcycle. Reprinted from The Design of Everyday Things (p.83), by D. A. Norman, 2013, New York: Basic Books


Figure 4. Lego motorcycle in pieces. Reprinted from The Design of Everyday Things (p.83), by D. A. Norman, 2013, New York: Basic Books. Based on these two images, it is possible to discuss how multimodal social semiotics is at

work. This example is not necessarily a multimodal analysis of these images, but rather they are

used to illustrate certain concepts within the theory.

The “two central categories in social semiotic are sign and mode” (Bezemer & Kress,

2016, p. 16). Modes are culturally shaped resources for meaning making such as images, writing

etc., and signs are realized through the use of modes. Even daily objects such as clothing and

furniture can be considered as meaningful because of “their social making, the purposes of their

making and the regularity of their use in social life” (Kress, 2010, p. 80). In this case, lego pieces

are modes, which are used for meaning-making purposes. Each mode has potentials

(affordances) and limitations. When we see a lego piece, we understand they are used to create

an object by bringing all the pieces together. The affordances allow us to build something out of


the given pieces. Further, the affordances (the material pieces, their color, and shape) allow us to

build something in different combinations of lego pieces and color of different pieces help us

determine functions of certain parts but again, we are constrained by limitations, which have

been shaped culturally and socially. These constraints prove to be helpful when constructing lego

pieces into a meaningful object. Norman (2013) talks about some constraints, among which are

semantic and cultural constraints. For instance, while constructing these pieces, there is only one

meaningful location for the rider, who must sit facing forward (Norman, 2013). Similarly, the

windshield is there to protect a rider’s face from the wind, so it must be in front of the rider

(Norman, 2013). How do we know this information? Because the world around us and the

context determine how we perceive different signs and how they relate to each other in context.

Culture plays an important role in our understanding of meanings. For instance, the word police

have to be placed right side up due to cultural constraints (Norman, 2013), or the cultural

constraints determine the actual places for the lights (traffic lights) and where they can be placed

because of their colors. Red is usually known as the stop light, so it should be placed in the rear.

White (also yellow) is used for headlights, so it would be wise to place it in the front of the

police motorcycle. Blue flashlight goes at the top of the car.

How are these messages are communicated to us? Cultural and social knowledge of the

world help us understand these messages, but sign makers or designers of the lego pieces are the

creators of this semiotic ensemble. “The process of sign making is always subject to the

availability of semiotic resources and to the aptness of the resources to the meanings that the sign

maker wishes to realize” (Bezemer & Kress, 2008, p. 170). Aptness refers to the idea that “the

form has requisite features to be the carrier of the meaning” (Kress, 2010, p.55). In this case,

designers decided to form raw materials into shapes that resemble to a human figure riding a


motorcycle to make a toy. Designers used lego material for that purpose but in another context, a

drawing or even a child’s drawing can fulfill the desire of sign-making if the sign-maker regards

features of their design as apt. The goal of sign-makers is to inform sign-makers about the

signified through the use of signs. Designers or sign-makers also use framing frame or set

boundaries to interpretation of the signs they design. A frame defines a world to be engaged, and

what is in a frame is separated from what is outside of a frame (Kress, 2010) chosen by the sign-

maker. In some cases, it is even possible to see a sign-maker’s stance in the world through

interpreting the signs. In this case of a lego toy, the audience only see a police officer riding a

motorcycle. However, if the object was put into another background, framing would have been

different – maybe contextually richer.

Makers of signs are also affected by the communities and histories of the societies that

they live in. Creating a lego toy might be a foreign concept to many cultures around the world.

At the same time, the audience’s interest in the object are marked by what they take as criterial

about a specific sign, the availability, materiality, and the affordances of modes (Kress, 2010).

The affordances of this lego police car might become a part of cultural and social habituation to

modes; in other words, a younger kid might be excited to see a lego toy for the first time, but in

habituation process, it becomes a normal part of our lives. The cycle of semiosis and meaning-

making continues in this way.

In time that habituation to representation can begin to shape our expectations about how we will encounter and engage with the world which we then represent. (Kress, 2010, p. 76)


Conclusion

Kress and van Leeuwen’s comprehensive grammar of visual design based on multimodal

social semiotics (1996, 2006) and later works paved the way for a new understanding of

meaning-making practices in media rich environments. However, one important point is that

social semiotics as a theory comes into life when it is applied to problems and instances in real

world, and it requires immersing oneself in semiotics concepts and methods as well as other

fields (van Leeuwen, 2005). Multimodal social semiotics can apply to any multimodal

communication setting, and there have already been studies which used the approach as an

analytic lens to better understand design choices and meaning-making through design (Connors,

2013, Jewitt, 2013; Machin, 2016; Michelson & Alvarez, 2016; Zhao, Djonov, & van Leeuwen,

2014). Understanding the components of such meaning-making practices in a semiotic level

would equip us with richer knowledge of the world around us and multimodal design practices.


References

Baldwin, N. (2016, December 19). What Is a Design Language…really? Retrieved from

https://medium.com/thinking-design/what-is-a-design-language-really-cd1ef87be793

Bezemer, J., & Jewitt, C. (2009). Multimodal analysis: Key issues. In L. Litosseliti (Ed.),

Research methods in linguistics (180-197). London & New York: Continuum.

Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of

designs for learning. Written Communication, 25(2), 166-195. doi:

10.1177/0741088307313177

Bezemer, J., & Kress, G. (2016). Multimodality, learning and communication: A social semiotic

frame. New York, NY: Routledge.

Buchanan, R. (1992). Wicked problems in design thinking. Design Issues, 8(2), 5-21. doi:

10.2307/1511637

Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge, MA: MIT Press.

Connors, S. P. (2013). Weaving multimodal meaning in a graphic novel reading group. Visual

Communication, 12(1), 27-53.

Cooper, R., Junginger, S., & Lockwood, T. (2009). Design thinking and design management: A

research and practice perspective. Design Management Review, 20(2), 46-55.

Halliday, M.A.K. (1978). Language as social semiotic. London: Edward Arnold. Halliday, M.A.K. (2003). Introduction: On the ‘architecture’ of human language. In J. Webster

(Ed.), On language and linguistics (pp. 1–29). London & New York: Continuum.


Halliday, M. A. K., & Matthiessen, C. M. (2013). Halliday's introduction to functional grammar.

New York, NY: Routledge.

Haratyan, F. (2011). Halliday’s SFL and social meaning. In 2nd International Conference on

Humanities, Historical and Social Sciences IPEDR, 17, 260-264.

Eco, U. (1984). The role of the reader: Explorations in the semiotic of

texts. London: Hutchinson.

Iedema, R. (2003). Multimodality, resemiotization: Extending the analysis of discourse as multi-

semiotic practice. Visual Communication, 2(1), 29-57. doi:

doi.org/10.1177/1470357203002001751

Jappy, T. (2013). Introduction to Peircean visual semiotics. New York & London: Bloomsbury.

Jewitt, C. (2006). Technology, literacy, learning: A multimodal approach. London: Routledge.

Jewitt, C. (2013). Multimodal methods for researching digital technologies. In S. Price, C. Jewitt,

& B. Brown (Eds.), The Sage handbook of digital technology research (pp. 250-265).

London: Sage.

Kress, G. (1976). Introduction. In G. Kress (Ed.), Halliday: System and function in language,

(pp. vii–xxi). Oxford: Oxford University Press.

Kress, G. (1997). Before writing: Rethinking the paths to literacy. London: Routledge.

Kress, G. (1998). Visual and verbal modes of representation in electronically mediated

communication: The potentials of new forms of text. In I. Snyder & M. Joyce (Eds.),

Page to screen: Taking literacy in the electronic era (pp. 53–79). Sydney, Australia:

Allen and Unwin.

Kress, G. (2003). Literacy in the new media age. London: Routledge.


Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication.

New York, NY: Routledge.

Kress, G. (2014). The rhetorical work of shaping the semiotic world. In A. Archer & D.

Newfield (Eds.), Multimodal approaches to research and pedagogy: Recognition,

resources, and access (pp. 131–152). New York, NY: Routledge/Taylor & Francis.

Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of graphic design.

London: Routledge.

Kress, G., & van Leeuwen, T. (1998). Front pages: The (critical) analysis of newspaper layout.

In A. Bell and P. Garrett (Eds.), Approaches to media discourse (pp. 186–219). London:

Blackwell.

Kress, G., & van Leeuwen, T. (2006). Visual interaction. In A. Jaworski & N. Coupland (Eds.),

The discourse reader (2nd ed., pp. 362–384). New York, NY: Routledge.

van Leeuwen, T. (2005). Introducing social semiotics. London & New York: Routledge.

Machin, D. (2016). Introduction to multimodal analysis. London & New York: Bloomsbury

Publishing.

Manghani, Piper, & Simons, J. (Eds.). (2006). Images: A reader. Thousand Oaks, CA: Sage.

Michelson, K., & Alvarez Valencia, J. A. (2016). Study abroad: Tourism or education? A

multimodal social semiotic analysis of institutional discourses of a promotional

website. Discourse & Communication, 10(3), 235-256.

Muntigl, P., & Ventola, E. (2010). Grammar: A neglected resource in interaction analysis? In J.

Streeck (Ed.), New adventures in language and interaction (pp. 99–124). Amsterdam,

Netherlands: John Benjamins.


Nordquist, R. (2018, July 16). Systemic Functional Linguistics. Retrieved from

https://www.thoughtco.com/systemic-functional-linguistics-1692022

Norman, D. A. (2013). The design of everyday things. New York: Basic Books.

Nöth, W., & Jungk, I. (2015). Peircean visual semiotics: Potentials to be

explored. Semiotica, 2015(207), 657-673. doi: 10.1515/sem-2015-0058

Piattelli-Palmarini, M. (1980). Language and learning: The debate between Jean Piaget and

Noam Chomsky. Cambridge, MA: Harvard University Press.

Saussure, F. D. (1916). Course in general linguistics (Cours de linguistique

générale). Translated by Wade Baskin. London: Peter Owen.

Vandendorpe, C. (2009). From papyrus to hypertext: Toward the universal digital library.

Champaign, Illinois: University of Illinois Press.

Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: A

multimodal social semiotic approach to PowerPoint. Text & Talk, 34(3), 349-375.

a social semiotic approach to multimedia: the work of ......theory of multimodal social semiotics...

Documents