semiotics in media frank nack (isla - uva)

Semiotics in Media Frank Nack (ISLA - UvA)

Semiotics - Definition

Semiotics is the study of sign processes (semiosis), signs and symbols, or signification and communication. It is usually divided into the three following branches:

• Semantics: Relation between signs and the things to which they refer

• Syntactics: Relations among signs in formal structures • Pragmatics: Relation between signs and their effects on

the people who use them

A fragment of semiotic history

Image by Göran Sonesson: The semiotic function and the genesis of pictorial meaning, http://www.arthist.lu.se/kultsem/sonesson/ImatraCourseTx1.html

Context of pictorial semiotics

Image by Göran Sonesson: The semiotic function and the genesis of pictorial meaning, http://www.arthist.lu.se/kultsem/sonesson/ImatraCourseTx1.html

Communication – or the creation of meaning

Fundamentals of Communication

Signal Source Signal Observer

p c

p = perceive c = conceive

Communication chain

Expedient Percipient

p c p c

Communication

•  is a process of transferring information from one entity to another •  is sign-mediated interaction between at least two agents •  both agents share a repertoire of signs and semiotic rules.

Communication chain

p c p c

Sign Repertoires

Description chain

p c p p

Reality Description

Query chain

Describer Archiver Enquirer

Communication semantics

Task knowledge

Expedient

structures Shared cultural

Shared social structures

Personality attributes

Organismic attributes e.g. male, adult, etc.

Outside cultural attributes

Outside social attributes

Percipient Media

Expression knowledge

Thematic knowledge

Meaning

Media semiotics

Text Image Video Audio Tactile

Semiotic concepts

14

The Sign - Saussure

Concept

Mental Perception of Media

SIGN

Signifier Signified beauty

15

The Sign - Peirce

SIGN

Representamen (symbol)

Object (referent)

Interpretant (thought)

active process

physical or mental entity?

psychological or ontological status?

referred to on a particular occasion? typical or ideal representation?

16

The Sign - Arbitrariness

Conventional in the Saussurean sense means that the relationship between the signifier and the signified dependents on social and cultural conventions.

The Saussurean model supports the notion of arbitrariness of the sign by proposing the autonomy of language in relation to reality. Its emphasis on internal structures within a sign system assumes that language does not “reflect” reality but rather constructs it.

Semantics – Index, Icon Symbol (Peirce)

Icon A sign which represents its object mainly through its similarity with some properties of the object, based on the reproduction of perceptual conditions.

Symbol A sign with an arbitrary link to its object (the representation is based on convention).

Index A sign which represents its object by an inherent relationship.

Semiotics – Text, Image, Video

Text – a sign system

Approaching text

A (Alphabet)‏

Saussure, Ferdinand de - (1857-1913) Swiss linguist. His Course in General Linguistics (1916, posthumous) is generally considered to be the foundation of modern linguistics. He envisaged the development of semiology as a science of signs.

Peirce, Charles S. - (1839-1914) American scientist and philosopher. One of the foremost philosophers of 'pragmatism' - no object or concept possesses validity or importance in its own right. Its significance lies only in the practical effects of its use or application. For Communication and Media students, his importance lies primarily in his development of semiotics.

B (Logogram)‏

Approaching text

Encoding - decoding

Encoding : framework of knowledge relation of production technical infrastructure

Decoding : syntactic recognition of the sign (in relation to other signs); semantic comprehension of the intended meaning of the sign; pragmatic interpretation of the sign in terms of relevance, agreement etc.

dominant reading

negotiated reading

oppositional reading

p c p c Sign

Text – a sign system I

Text – a sign system II

Syntagms are often defined as 'sequential' (and thus temporal - as in speech and music), but they can represent spatial relationships. The plane of the syntagm is that of the combination of 'this-and-this-and-this' (syntax).

A paradigmatic structure represents potential substitutions in which a range of candidates can take the place of a sign in the syntagmatic structure. The plane of the paradigm is that of the selection of 'this-or-this-or-this' (semantics).

shoes socks pants sweater scarf hat

knickers short

kilt tights

Example:

Text – a sign system III

Syntagm •  Spatial relations (horizontal and vertical axi, centre and margin)‏ •  Logical order (grammar)‏ •  Exposition (proposition, evidence, justification •  Narrative space (exposition, retardation, digression, omission, redundancy)‏ •  Narrative time (ellipses, compression, insertion, dilation)‏ Paradigm •  clusters (e.g. synonyms)‏ •  doublets (e.g. oppositions)‏ •  proportional series ( e.g. a series of oppositional doublets such as female - male, passive - active, etc.)‏ => Taxonomy •  hierarchies (ordered semantic units based on relations of inclusion or exclusion, e.g. Pekinese/dog/animal/living thing). => Thesaurus

Semantic field: '...a conceptual structure which organises potential meanings in relation to others' => Conceptual graph, semantic network, ontology

processes

Representation and Transformation mechanisms

Text – a sign system IV

A code is a rule-governed system of signs, whose rules and conventions are shared amongst members of a culture, and which is used to generate and circulate meanings in and for that culture. Fiske, J. (1987, p.4)

A set of signs that carry meaning.

A set of agreed rules for combining those signs together

  Perceptual (e.g. Typography)‏   Syntagmatic (e.g. Grammar)‏   Paradigmatic (e.g. Ontology)‏   Social (e.g. Word use)‏

Text – a sign system summary I

Text is a sign system strong on arbitrariness, proposing the autonomy of language in relation to reality.

Text emphasis on internal structures and thus does not 'reflect' reality but rather constructs it.

Text is conventional with an emphasis on the types index and symbol.

Text – a sign system summary II

Representing Text in a media-based system:

Conceptual models for:  Typography  Layout  Writing system (e.g. Alphabet)‏  Syntax (e.g. grammar, markup languages, ....)‏  Dictionaries  Semantics (e.g. taxonomy, thesaurus, ontology, conceptual graph, etc.)‏  Style (e.g. frame, template, script,....)‏  Genre (e.g. template, conceptual graph)‏

Interpretation depends on the task:  Search (e.g. text understanding, word matching and/or ranking)‏  Generation (e.g. text understanding, question- answering, ....)‏  Comparison (e.g. Syntax (pattern matching) or semantics (clustering, distance evaluation, etc.)‏

Text – Example use

Twitoems

Syntagma

Syntagma + IPA (code)

Aesthetic codes Rhetorical codes Stylistic codes

Perceptional Codes

29

Image – a sign system

" Legend of Orpheus & Eurydice ", 2001, The Werner Collection http://www.wernercollection.com/WorldView1.htm

Approaching an image

Concept

Mental perception of media

SIGN

Signifier (material)‏

Signified (meaning)‏ ?

31

Approaching an image

Colour Object

Mise en scene Framing Genre

Meaning

Materiality

Distance (foreground - background)‏

32

Image – a sign system I

Perceptual codes •  perceptive codes (establish the condition for effective perception) •  recognition codes which are blocks of signifieds we use to recognize objects •  transmission codes which construct the determining conditions for the perception of an image (dots that make up a newspaper image)

Textual codes •  tonal codes address the prosodic features by connoting them with particular intonation of the sign •  Iconic codes (figures, signs, semes) •  Iconographic codes connote more complex and culturalized semes that are immediately identifiable and classifiable, such as "the four horsemen of the Apocalypse".

33

Image – a sign system II

Social codes •  verbal language •  bodily codes (bodily contact, physical orientation, gaze, gestures and posture); •  commodity codes (fashions, clothing, cars); •  behavioural codes (protocols, rituals, role-playing, games) •  ideological codes (encoding' and 'decoding' information by using theories such as individualism, liberalism, feminism, materialism, capitalism, socialism, etc.)

Syntagmatic - paradigmatic codes •  scientific codes, including mathematics; •  aesthetic codes (poetry, drama, painting, sculpture, music, etc.) •  genre, rhetorical and stylistic codes (e.g. in narrative: plot, character, action, dialogue, setting, etc.), •  mass media codes (e.g. in photography, TV, film, radio, newspaper and magazine, etc.)

34

Image – a sign system III

Denotation describes the 'literal' or 'obvious' meaning of a sign. Thus, denotation of a representational visual image is what all viewers from any culture and at any time would recognize the image as depicting.

Denotation is the first level of signification.

Perceptual codes Textual codes Social codes

Sensory system Media

Sign I (denotative sign with signifier and signified)‏

35

Image – a sign system IV

Connotation refers to the socio-cultural and 'personal' associations (ideological, emotional etc.) of the sign. These are typically related to the interpreter's class, age, gender, ethnicity and so on.

Connotation is the second level of signification. Sign I + Signified

Social codes Syntagmatic codes + experiences Paradigmatic codes associations

Sign II (connotative sign with signifier and signified)‏

36

Image – a sign system VI

The third level of signification.

Sign III (valued signifier and signified)‏

Sign II Social codes Syntagmatic codes Paradigmatic codes

value

37

Image – a sign system IV

Signification difference between text and image

On the 1st step: Text => provides an index as a signified Image => sets the signified

The reader replaces each index (word) of the provided order with a signified for his or her liking => the 3rd step of signification does not cause a problem, as it is already matched in the first step

The viewer has to establish the order of importance (using step 2) => particular the 3rd step of signification becomes important as it is the comparison with the own sign system (comparison with what is not shown) that determines how the perceiver values and thus understands the material.

38

Image – Description methods

" Legend of Orpheus & Eurydice ", 2001, The Werner Collection

39

Image – a sign system summary

An image is a a dominantly iconic sign system, proposing a union in relation to reality.

The denotative power of an image, the optical pattern, communicates a precise knowledge, which releases the audience from the process of decision making but leaves a problem of interpretation (signification process).

40

Image – a sign system summary II

Representing an Image in a media-based system:

Conceptual models for:  quantitative or qualitative characterization of optical pattern (feature extraction (colour, texture, light, angle, etc.), pattern recognition (line, shape region, etc.), multi-scale signal analysis, ...)‏  Spatial dimensions => textual metadata  Semantics (e.g. taxonomy, thesaurus, ontology, etc.)‏  Semantic markers (key word, tag, schema, ....)‏ to express higher semantics , such as forms, styles, genres, aesthetics, social codes.

Interpretation depends on the task:  Search (e.g. retrieval by example)‏  Generation (e.g. Qualitative support on features and higher semantics)‏  Presentation (e.g. browsing through collage)‏  Automatic art generation

Image – Example use

Semiotic tagging (collaboration with Ansgar Scherp, West, University Koblenz-Landau)

Icon Index Symbol

Classification of ‘tags’ while annotating

Identifying the meaning of the image in the query based on classification slots

Step A

Step B

Rhetorical tropes

Image from Luis Buñuel’s “Un Chien Andalou”

Rhetorical tropes

A trope describes the use of a word or expression as changed from the original signification to another. Tropes are mainly of four kinds:

Trope

metaphor,

metonymy,

synecdoche,

and irony.

Rhetorical tropes

A metaphor involves one signified acting as a signifier referring to a different signified.

A metaphor is a figure of speech in which an expression is used to refer to something that it does not literally denote in order to suggest a similarity.

Metaphor

Rhetorical tropes

Metonymy

Metonymy is a function which involves using one signified to stand for another signified which is directly related to it or closely associated with it in some way. (Chandler, http://www.aber.ac.uk/media/Documents/S4B/sem07.html)

Metonyms are based on various indexical relationships between signifieds, notably the substitution of

•  effect for cause •  object for user •  substance for form •  place for event •  place for person •  place for institution •  institution for people

See also: Catherine R. Langan’s article “Intertextuality in Advertisements for Silk Cut Cigarettes”, available at http://www.aber.ac.uk/media/Students/crl9502.html

Rhetorical tropes

Synecdoche

A synecdoche substitutes a part of a thing for the whole, or the whole for a part, the species for the genus or vice versa.

Examples: •  part for whole (fifty sails for fifty ships ) •  whole for part (the market for customers) •  species for genus - hypernymy (cutthroat for assassin) •  genus for species - hyponymy (vehicle for car).

In visual media a close-up is a simple synecdoche - a part representing the whole.

(Jakobson & Halle 1956, p. 92)

Image from Jean-Luc Godard's ‘La Chinoise'

Rhetorical tropes

Irony

Irony involves incongruity between what is expected and what occurs.

Irony involves a shift in modality. The evaluation of the ironic sign requires the retrospective assessment of its modality status. Re-evaluating an apparently literal sign for ironic cues requires reference to perceived intent and to truth status. Thus, an ironic sign is 'double-coded'.

The signifier of the ironic sign seems to signify one thing but another signifier establishes that it actually signifies something very different. Where it means the opposite of what it says (as it usually does) it is based on binary opposition.

48

Video – a temporal visual sign system

Video – Examples for content repurpose

The shining – a romantic comedy

Titanic – The sequel

Schindler's list – the romantic comedy

Mary Poppins – the scary original

All images from YouTube (www.youtube.com)‏

50

Video – Internal and external context

An image is an index to a story

Images from Stanley Kubrick's 'Shining'

A video is an iconic representation of a story

51


Media-dependent context

  Spatial   Temporal

Meta-semantic context

  Plot structure   Genre   References   Reviews   Personal preferences  ....


Video – Media-dependent context - temporal

t

Shot context: temporal relationship between frames

Scene context: temporal relationship between shots, e.g. insert

Episode context: temporal relationship between scenes

Montage Metric (absolute length)‏ Rhythmic Tonal

Compression Expansion Insertion Deletion


Temporal

Spatial

t

Video – Media-dependent context - summary

All frames from “The Shining” (1980)‏

Knowledge representation

54


Media-dependent context

  Spatial   Temporal

Meta-semantic context

  Plot structure   Genre   References   Reviews   Personal preferences  ....


Video – Meta-Semantic context - Narration III

Characteristic objects and actions from the real world

Fabula

Style

Genre a loose set of criteria for a category of composition

Theme as a set of possible narrative objects (comedy, tragedy, etc.)‏

Plot

Structure Catalyst – Conflict -

Consequence


Video – Meta-Semantic context - Narration IV


Syntagma in video

Autonomous sequence

Metz's syntagmatic categories (Monaco, 1981, p. 188).

Autonomous shot (establishing shot, insert)

Syntagms

Parallel syntagm (montage of motifs)

Chronological syntagms

Achronological syntagms Bracket syntagm (montage of brief shots)

Descriptive syntagm (sequence describing one moment)

Narrative syntagms

Alternative narrative syntagm (two sequences alternating)

Linear narrative syntagms

Scene (shots implying temporal continuity)

Sequences proper

Episodic sequence (organized discontinuity of shots)

Ordinary sequence (temporal with some compression)

Syntagmatic structures of construction (Monaco, 1981, p. 145).

Space (Frame)

Time (shot, scene, sequence)

Construction Syntagms

Syntagms and paradigms

Paradigm - Examples

shoes socks pants sweater scarf hat

Paradigmatic structures of clothing (Monaco, 1981, p. 341).

skirt knickers short

kilt culottes tights

Choice process example in film: Casting Image from Charlie Kaufman’s ’Synecdoche New York'

59

Video – Context - summary

MPEG - 1 AVI

Video

Time

60

Video – a temporal visual sign system - summary

 A video is a compositional unit with individualised semantics.

 The semantics may change if a shot is juxtaposed with another shot.

 A distinction between filmic (codify the relation to reality) and cinematic codes (codify narrative communication) must be made.

 Video, though based on common human content and thematic structures, provides its own realities of time and space which are interwoven in the narrative structure.

 A story is a representational system based on two main layers, structure and content, each serving two distinct purposes (form and substance) simultaneously.

Video – Example use

AUTEUR (1996)

•  well described video components •  generation of humorous sequences from clip database •  rules for continuity editing, action generation, themes, and presentation generation

Interactive Production Generator – Narrative Model

Nack (1996)

AUTEUR - Aim

•  Automatic composition of visual slapstick sequences.

•  Use an existing, arbitrary data base.

•  Investigate machine creativity.

AUTEUR – Generation Example I

Motivation

Narrative rules (joke structure)‏ Access via content representation

Editing rules (continuity)‏

Narrative rules (continuity)‏ Access via content representation

Editing rules (continuity) ‏

Realisation

Narrative rules (joke structure)‏ Access via content representation Editing rules (continuity)‏

AUTEUR – Generation Example II

Resolution

Narrative rules (joke structure)‏ Access via content representation Editing rules (continuity)‏

AUTEUR – Generation Example III

Result

AUTEUR – Generation Example IV


Narranotations

An approach for an association-based story environment, in which a priori unrelated experiences represented in images, are stitched together to guide users through interesting city spaces.

Interesting spaces are described as ‘hypespot’, which facilitates linking the real world with the structure of the story.

Narranotations (annotation to the image) provide information about how an expression can be used as an element within a story.


N a r r a n o t a t I o n s


N a r r a n o t a t i o n s

Discussion

Image from Charlie Kaufman’s ’Synecdoche New York'

References

Image from Charlie Kaufman’s ’Synecdoche New York'

Bibliography - Fundamentals

  Andersen, P. B. (1997). Semiotic Approaches to Constrution and Assessment Of Computer Systems. Cambridge Universty Press

  Barthes, R. Image Music Text. Hill and Wang, New York, 1977.

  Bolter, J. G. & Grusin, R. (1999). Remediation - Understanding New Media. Cambridge, Massachusetts: MIT Press.

  Chandler, D. (2003). Semiotics for Beginners - http://www.aber.ac.uk/media/Documents/S4B/semiotic.html

  Corner, John (1980). 'Codes and Cultural Analysis', Media, Culture and Society 2: 73-86 Davis, H. & Walton P. (Eds.) (1983). Language, Image, Media. Oxford: Basil Blackwell

  Derrida, J. Writing and Difference. Routledge & Kegan Paul, London, 1978.

  Ducrot, O. and Todorov, T. Encyclopedic Dictionary of the Sciences of Language. The Johns Hopkins University Press, Baltimore, Maryland, 1979.


  Eco, U. A Theory of Semiotics. Indiana University Press, Bloomington, Indiana, 1976.

  Fiske, J. (1982). Introduction to Communication Studies. London: Routledge

  Greimas, J. (1983). Structural Semantics: An Attempt at a Method. Lincoln: University of Nebraska Press.

  Gumpert, G. & Cathcar, R. (1985). Media Grammars, Generations and Media Gaps. Critical Studies in Mass Communication 2: 23-35

  Hawkes, T. Structuralism and Semiotics. University of California Press, Berkeley, 1977.

  Hodge, R. & Kress, G. (1988): Social Semiotics. Cambridge: Polity

  Kendon, A. (1981). Nonverbal Communication, Interaction, and Gesture - Selections from Semiotica. The Hague: Mouton Publishers.

  Lévi-Strauss, C. (1968). Structural Anthropology Volume 1, translated by Clair Jacobson, Brooke Grundfest Schoepf. London: Allen Lane.

  Lévi-Strauss, C. (1977). Structural Anthropology Volume 2, translated by Monique Layton. London: Allen Lane.

  Lippmann, W. (1934). Public Opinion. New York: The MacMillan Company.


  McLuhan, M. The Gutenberg Galaxy: The Making of Typographic Man. University of Toronto Press, Toronto, 1962.

  McLuhan, M. (1962). The Gutenberg Galaxy. Toronto: University of Toronto Press

  McLuhan, M. & Quentin Fiore (1967). The Medium is the Massage. New York: Bantam

  McLuhan, M. (1970). Counterblast. London: Rapp & Whiting

  McLuhan, M. Understanding Media: The Extensions of Man. The MIT Press, Cambridge, Massachusetts, 1994.

  Metz, C. Film Language: A Semiotics of Cinema. The University of Chicago Press, Chicago, 1974.

  Mick, D. G. (1986): 'Consumer Research and Semiotics: Exploring the Morphology of Signs, Symbols and Significance', Journal of Consumer Research 13(2): 196-213

  Minsky, M. L. (1988). The Society of mind. London: Picador.

  Nöth, W. (1990). Handbook of Semiotics. Bloomington, IN: Indiana University Press


  Packer, R. & Jordan, K. (editors) (2001). multiMEDIA - From Wagner to Virtual Reality. New York, W.W.Norton & Company

  Peirce, C. S. (1960). The Collected Papers of Charles Sanders Peirce - 1 Principles of Philosophy and 2 Elements of Logic, Edited by Charles Hartshorne and Paul Weiss. Cambridge, Massachusetts: The Belknap Press of Harvard University Press.

  Reddy, M. The Conduit Metaphor: A Case of Frame Conflict in Our Language about Language. in Ortony, A. ed. Metaphor and Thought, Cambridge University Press, New York, 1979, 284-324.

  Saussure, F.d. Course in General Linguistics. McGraw-Hill, New York, 1983.

  Sebeok, T. A. & Danesi, M. (2000). The Forms of Meaning – Modeling Systsems Theory and Semiotic Analysis, Mouton de Gryter, Berlin

  Sowa, J. F. (1984). Conceptual Structures: Information Processing in Mind and Machine. Reading, MA: Addison-Wesley Publishing Company.

  Stam, R., Burgoyne, R. and Flitterman-Lewis, S. New Vocabularies in Film Semiotics: Structuralism, Post-Structuralism, and Beyond. Routledge, London, 1992.


  Sturrock, J. (1986): Structuralism. London: Paladin

  Todorov, T. Theories of the Symbol. Cornell University Press, 1977.

  Wilden, A. (1987). The Rules Are No Game: The Strategy of Communication. London: Routledge & Kegan Paul

  Wolff, C. (1972). A Psychology of Gesture. New York: Arno Press.

Bibliography - Text

  Ball, M. (1997). Narratology - Introduction to the Theory of Narrative (2nd edition). Toronto: University of Toronto Press.

  Barrett, E. (ed.), The Society of Text: Hypertext, Hypermedia, and the Social Construction of Information. The MIT Press, Cambridge, Massachusetts, 1989.

  Barrett, E. (ed.), Text, ConText, and HyperText: Writing with and for the Computer. The MIT Press, Cambridge, Massachusetts, 1988.

  Barthes, R. Elements of Semiology. Hill and Wang, New York, 1967.

  Barthes, R. (1974). S/Z with a preface by Richard Howard. London: Jonathan Cape.

  Barthes, R. (1977). Image, Music, Text - Essays selected and translated by Stephen Heath. London: Fontana Press.

  Bernstein, M., Patterns of Hypertext. in ACM Hypertext'98, (Pittsburg, Pennsylvania, 1998), ACM Press, 21-29.

  Berwick, R.C. Computational Structure of Language.

Bibliography - Text

  Berwick, R.C. and Weinberg, A.S. The Grammatical Basis of Linguistic Performance: Language Use and Acquisition. MIT Press, Cambridge, Mass., 1984.

  Bordwell, D. (1989). Making Meaning - Inference and Rhetoric in the Interpretation of Cinema. Cambridge, Massachusetts: Harvard University Press.

  Brady, M., Berwick, R.C. and Allen, J. Computational Models of Discourse. MIT Press, Cambridge, Massachusetts, 1983.

  Brusilovsky, P., Kobsa, A., and Vassileva J.(ed.) Adaptive Hypertext and Hypermedia. Kluwer Academic Publishers, Dordrecht, The Netherlands, 1998.

  Bush, V. As We May Think. Atlantic Monthly, 176 (July). 101-108.

  Chatman, S.B. Story and Discourse: Narrative Structure in Fiction and Film. Cornell University Press, Ithaca, N.Y., 1978.

  Chatman, S.B. Coming to Terms – The Rhetoric of Narrative in Fiction and Film. Cornell University Press, Ithaca, N.Y., 1990.

Bibliography - Text

  Chomsky, N. Aspects of the Theory of Syntax. The MIT Press, Cambridge, 1965.

  Chomsky, N. Language and Mind. Harcourt Brace & World, Inc, New York, 1968.

  Chomsky, N. Studies on Semantics in Generative Grammar. Mouton, The Hague, 1972. Chomsky, N. Rules and Representations, 1980.

  Chomsky, N. Lectures on Government and Binding: The Pisa Lectures. Foris Publications, Holland, 1981.

  Chomsky, N. The Logical Structure of Linguistic Theory. University of Chicago Press, Chicago, 1985.

  Chomsky, N. Language and Thought. Moyer Bell, Wakefield, Rhode Island, 1993.

  Chomsky, N. On Language: Chomsky's Classic Works "Language and Responsibility" and "Reflections on Language" in One Volume. The New Press, New York, 1998.

Bibliography - Text

  Delany, P. and Landow, G.P. Hypermedia and Literary Studies, 1991.

  Eco, U. (1966). 'Narrative Structure in Fleming'. In E. del Buono & Umberto Eco (Eds.): The Bond Affair. London: Macdonald

  Eco, U. (1981). The Role of the Reader. London: Hutchinson

  Eco, U. (1984). Semiotics and the Philosophy of Language. Bloomington, IN: Indiana University Press

  Engelbart, D. A Conceptual Framework for the Augmentation of Human Intellect. in Howerton, P.D., Weeks, D.C ed. Vistas in Information Handling, Spartan Books, Washington DC, 1963.

  Goodman, N. Languages of Art. Hackett Publishing Company, Inc., 1976.

  Greimas, A.J. Structural Semantics: An Attempt at a Method. University of Nebraska Press, Lincoln, 1983.

  Holdcroft, D. (1991). Saussure: Signs, Systems and Arbitrariness. Cambridge: Cambridge University Press

  Illich, I. and Sanders, B. ABC: The Alphabetization of the Popular Mind. North Point Press, San Francisco, 1988.

Bibliography - Text

  Iser, W. (1989). Prospecting: From Reader Response in Literary Anthropology. Baltimore, Maryland: The John Hopkins University Press.

  Iser, W. (1993). The Fictive and the Imaginary: Charting Literary Anthropology. Baltimore, Maryland: The John Hopkins University Press.

  Jakobson, R. (1971): 'Language in Relation to Other Communication Systems'. In Roman Jakobson (Ed.): Selected Writings, Vol. 2. Mouton: The Hague, pp. 570-79

  Jakobson, R.H., M. Fundamentals of Language. Mouton Publishers, The Hague, 1980.

  Kolb, D. Socrates in the Labyrinth: Hypertext, Argument, Philosophy. Eastfgate Systems, Cambridge, 1995.

  Kumpf, E.P. Visual Metadiscourse: Designing the Considerate Text. Technical Communication Quarterly, 9 (4). 401-424.

  Lakoff, G. P. (1972). Structural complexity in fairy tales. The study of man, 1, 128 - 190.

  Lakoff, G. & Johnson, M. (1980). Metaphors We Live By. Chicago: University of Chicago Press

Bibliography - Text

  Leiss, W., Kline, S. & Jhally, S. (1990). Social Communication in Advertising: Persons, Products and Images of Well-Being (2nd Edn.). London: Routledge

  Lemon, L. T., & Reis, M. J. (1965). Russian Formalist Criticism - Four Essays. Lincoln: University of Nebraska Press.

  McQuarrie, E. F. & Mick, D. G. (1999). Visual Rhetoric in Advertising: Text-Interpretative, Experimental and Reader-Response Analyses, Journal of Consumer Research 26(1): 37-54

  Miles, A. Hypertext Syntagmas: Cinematic Narration with Links. Journal of Digital Information, 1 (7).

  Moulthrop, S., Beyond the Electronic Book: A Critique of Hypertext Rhetoric. in 4th ACM conference on Hypertext, (San Antonio, USA, 1991), 291 - 298.

  Moulthrop, S. and Cohen, S. The Color of Television, 1996, A multi-threaded, verbal/graphic Web fiction in collaboration with the writer and designer Sean Cohen.

  Nelson, T. Computer Lib and Dream Machines. Microsoft Press, Redmond, Washington, 1987.

  Nelson, T. Literary Machines, 1981.

Bibliography - Text

  Nielsen, J. Hypertext & Hypermedia. Academic Press, Inc.,, Boston, 1990.

  Nielson, J. Multimedia and Hypertext: The Internet and Beyond. Academic Press, Inc, Boston, 1990.

  Nyce, J.M. and Kahn, P. (eds.). From Memex to Hypertext: Vannevar Bush and the Mind's Machine. Academic Press, Boston, 1991.

  Ong, W. Interfaces of the Word. Cornell University Press, Ithaca, 1977.

  Ong, W. The Presence of the Word. Yale University Press, New Haven, 1967.

  Ong, W.J. Orality and Literacy: The Technologizing of the Word. Methuen, London, 1982.

  Price, G. (1973). A grammar of story: An introduction. The Hague: Mouton.

  Propp, V. W. (1968). Morphology of the Folktale. University of Texas Press.

  Raskin, V. (1985). Semantic Mechanisms of Humor. Dordrecht: D. Reidel Publishing Company.

  Ricoeur, P. (1985). Time and Narrative. Chicago: The University of Chicago Press.

Bibliography - Text

  Sampson, G. Writing Systems: A Linguistic Introduction. Stanford University Press, Stanford, California, 1985.

  Suleiman, S. R. (1983). Authoritarian Fictions: The Ideological Novel As a Literary Genre. New York: Columbia University Press.

  Suls, J. M. (1972). A two-stage model for the appreciation of jokes and cartoons: An information-processing analysis. In J. H. Goldstein & P. E. McGhee (Eds.), The Psychology of Humour (pp. 81 - 100). New York, London: Academic Press.

  Saussure, F. d. (1966). Course in General Linguistics - edited by Charles Balley, Albert Sechehaye and Albert Riedlinger. New York: McGraw-Hill.

  Segre, C. (1979). Structure and Time - Narration, Poetry, Models. Chicago: The University of Chicago Press.

  Tosca, S.P., A Pragmatics of Links. in 11th ACM conference on Hypertext and Hypermedia, (San Antonio, Texas, USA, 2000), pp. 77 - 84.

  Winograd, T. Language as a Cognitive Process: Syntax. Addison-Wesley, Massachusetts, 1983.

  Winograd, T. Understanding Natural Language. Academic Press, 1972.

Bibliography - Text

  Zellweger, P.T., Chang, B-W., and Mackinlay, J. D., Fluid Links for Informed and Incremental Link Transitions. in 9th ACM conference on Hypertext and Hypermedia, (Pittsburgh, USA, 1998), 50 - 57.

Bibliography - Image

  Arnheim, R. Art and Visual Perception: A Psychology of the Creative Eye. University of California Press, Berkeley, 1974.

  Arnheim, R. (1983). Film as Art. London: Faber & Faber. Argyle, M. (1988): Bodily Communication (2nd edn.). London: Methuen

  Aumont, J. The Image. British Film Institute, London, 1997.

  Barthes, R. Elements of Semiology. Hill and Wang, New York, 1967.

  Barthes, R. Image Music Text. Hill and Wang, New York, 1977.

  Barthes, R. S/Z, 1974.

  Bazin, A. (1967). The Ontology of the Photographic Image. In H. Gray (Eds.), What is Cinema - essays selected and translated by Hugh Gray (pp. 9 - 16). Berkeley: University of California Press.


  Cook, G. (1992). The Discourse of Advertising. London: Routledge

  Del Bimbo, A. (1999). "Visual Information Retrieval", Morgan Kaufmann Ed, San Francisco, USA

  Eco, U. (1982). Critique of the Image. In Burgin, V. (Ed.), Thinking Photography. London: Macmillan

  Forceville, C. (1996). Pictorial Metaphor in Advertising. London: Routledge

  Galtung, J. & Ruge, E. (1981). Structuring and Selecting News. In Cohen, S. & Young, J. (Eds.) (1981): The Manufacture of News: Social Problems, Deviance and the Mass Media. London: Constable ., pp. 52-63

  Gibson, J. J. (1950). Perception of the Visual World. Boston, Ma: Houghton Mifflin.

  Gibson, J. J. (1971). The Information available in Pictures. Leonardo(4), 27 - 35.


  Gombrich, E. H (1974). The Visual Image'. In David R Olson (Ed.): Media and Symbols: The Forms of Expression, Communication and Education. Chicago, IL: University of Chicago Press, pp. 255-8; first published in Scientific American 227 (September 1971): 82-96

  Gombrich, E. H (1977). Art and Illusion: A Study in the Psychology of Pictorial Representation. London: Phaidon

  Gombrich, E.H. Image and Code. Horace H. Rackham School of Graduate Studies, Ann Arbor, MI, 1981.

  Gombrich, E. H (1982). The Image and the Eye: Further Studies in the Psychology of Pictorial Representation. London: Phaidon

  Gombrich, E. H (1995). The Story of Art. London: Phaidon

  Goodman, N. Languages of Art. Hackett Publishing Company, Inc., 1976.

  Grayson, K. (1998). The Icons of Consumer Research: Using Signs to Represent Consumers' Reality. In Stern, B. B (Ed) (1998): Representing Consumers: Voices, Views and Visions. London: Routledge, pp. 27-43.


  Hall, S. (1981). The Determinations of News Photographs. In Cohen, S. & Young, J. (Eds.) (1981): The Manufacture of News: Social Problems, Deviance and the Mass Media. London: Constable , pp. 226-43

  Hochberg, J.E. Perception. Prentice-Hall, New Jersey, 1978.

  Kennedy, J.M. A Psychology of Picture Perception. Jossey-Bass, San Francisco, 1974.

  Kress, G. & van Leeuwen, T. (1996): Reading Images: The Grammar of Visual Design. London: Routledge

  Kress, G. & van Leeuwe, T. (1998): 'Front Pages: (The Critical) Analysis of Newspaper Layout'. In Allan Bell & Peter Garrett (Eds.): Approaches to Media Discourse. Oxford: Blackwell, pp. 186-219

  Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Freeman, San Francisco, 1982.

  McCloud, S. Understanding Comics: The Invisible Art. Tundra Publishing, Northampton, Massachusetts, 1993.


  McLuhan, M. Counter-Blast. Rapp & Whiting, London, 1970.

  McLuhan, M. The Gutenberg Galaxy: The Making of Typographic Man. University of Toronto Press, Toronto, 1962.

  McLuhan, M. and Fiore, Q. The Medium is the Massage. Bantam Books, New York, 1967.

  Panofsky, E. Meaning in the Visual Arts: Papers in and on Art History. Doubleday, Garden City, N.Y., 1955.

  Pentland, A.P., Picard, R.W. and Sclaroff, S. Photobook: Tools for Content-Based Manipulation of Image Databases, Massachusetts Institute of Technology Media Laboratory, 1994.

  Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A. and Jain, R. Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (12). 1349-1380.

  Sonesson, G. (1992). The semiotic function and the genesis of pictorial meaning, in Tarasti, Eero, ed.,Center/Periphery in representations and institutions. Proceedings from the 3rd Annual Meeting and Congress of The International Semiotics Insitute, Imatra, Finland, July 16-21, 1990.: International Semiotics Institute, Imatra 1992; ss 211-256.


  Strothotte, S.S., T. Seeing Between the Pixels. Springer, Berlin, 1997.

  Strothotte, T. Computational Visualization - Graphics, Abstraction and Interactivity. Springer, Berlin, 1998.

  Tudor, A. Image and Influence. George Allen & Unwin Ltd., London, 1974.

  Tufte, E.R. Envisioning Information. Graphics Press, Cheshire, Connecticut, 1990.

  Tufte, E.R. The Visual Display of Quantitative Information. Graphics Press, Cheshire, 1983.

  Tufte, E.R. Visual Explanations: Images And Quantities, Evidence And Narrative. Graphics Press, Cheshire, Connecticut, 1977.

Bibliography – Temporal Media

  Andrew, J.D. The Major Film Theories: An Introduction. Oxford University Press, Oxford, 1976.

  Andrew, D. Concepts in Film Theory. Oxford University Press, Oxford, 1984.

  Arnheim, R. Film as Art. Faber & Faber, London, 1983.

  Baggaley, J. & Duck S. (1976). Dynamics of Television. Farnborough: Saxon House

  Balázs, B. Theory of the Film: Character and Growth of a New Art. Dover Publications, Inc., 1970.

  Barthes, R. The Sequences of Actions. in Strelka, J. ed. Patterns of Literary Style, State University of Pennsylvania Press, University Park, Pennsylvania, 1971.

  Bazin, A. What is Cinema? (Volume I). University of California Press, Berkeley, 1971.

  Bazin, A. What is Cinema? (Volume II), 1971.


  Bellour, R. To Analyze, to Segment. Quarterly Review of Film Studies, 1, 3 (August). 331-354.

  Bettetini, G. The Language and Technique of the Film, 1973.

  Bloch, G.R. Elements d'une Machine de Montage Pour l'Audio-Visuel, Ecole Nationale Supérieure Des Télécommunications, 1986.

  Bloch, G.R. From Concepts to Film Sequences, Yale University Department of Computer Science, 1987.

  Bordwell, D., Staiger, J. and Thompson, K. The Classical Hollywood Cinema: Film Style and Mode of Production to 1960. Columbia University Press, New York, 1985.

  Bordwell, D. Narration in the Fiction Film. University of Wisconsin Press, Madison, 1985.

  Bordwell, D. Making Meaning: Inference and Rhetoric in the Interpretation of Cinema. Harvard University Press, Cambridge, Massachusetts, 1989.

  Bordwell, D. The Cinema of Eisenstein. Harvard University Press, Cambridge, Massachusetts, 1993.


  Bordwell, D. On the History of Film Style. Harvard University Press, Cambridge, Massachusetts, 1997.

  Bordwell, D. Planet Hong Kong: Popular Cinema and the Art of Entertainment. Harvard University Press, Cambridge, 2000.

  Bordwell, D. and Thompson, K. Film Art: An Introduction. McGraw Hill, New York, 2003.

  Burch, N. Theory of Film Practice. Princeton University Press, Princeton, 1969.

  Buckland, W. (2000). The Cognitive Semiotics of Film. Cambridge, UK: Cambridge University Press.

  Carroll, J.M. Toward a Structural Psychology of Cinema. Mouton Publishers, The Hague, 1980.

  Chatman, S.B. Story and Discourse: Narrative Structure in Fiction and Film. Cornell University Press, Ithaca, N.Y., 1978.

  Davis, D. (1960). The Grammar of Television Production (revised by John Elliot). London: Barrie & Rockliff


  Eco, U. (1965). 'Towards a Semiotic Enquiry into the Television Message', In Corner & Hawthorn (Eds.) (1980), pp. 131-50

  Eco, U. Articulations of the Cinematic Code. in Nichols, B. ed. Movies and Methods: An Anthology, University of California Press, Berkeley, 1976, 590-607.

  Eco, U. A Theory of Semiotics. Indiana University Press, Bloomington, Indiana, 1976.

  Eisenstein, S.M. The Film Sense. Harcourt Brace Jovanovich, Publishers, San Diego, 1947.

  Eisenstein, S.M. Film Form: Essays in Film Theory. Harcourt Brace Jovanovich, Publishers, San Diego, 1949.

  Eisenstein, S.M. Film Essays and a Lecture. Princeton University Press, Princeton, New Jersey, 1982.

  Eisenstein Selected Works: Writings 1922 - 1934. BFI Publishing, London, 1988.


  Eisenstein Selected Works: Towards a Theory of Montage. BFI Publishing, London, 1991.

  Fiske, J. (1987). Television Culture. London: Routledge

  Fiske, J. & John H. (1978): Reading Television. London: Methuen

  Frith, U., & Robson, J. E. (1975). Perceiving the language of film. Perception, 4, 97 - 103.

  Greenway, T. and Mouchawar, R., A Visual Language for the Management of Digital Film and Video (Motion Imagery) Archives. in 136th Technical Conference and World Media Expo of the Society of Motion Picture and Television Engineers (SMPTE), (Los Angeles, California, 1994), Society of Motion Picture and Television Engineers, Inc., 1-15.

  Gregory, J. R. (1961) Some Psychological Aspects of Motion Picture Montage. Ph.D. Thesis, University of Illinois.

  Gregory, R. L. (1971). The Intelligent Eye. London: Weidenfeld&Nicolson.


  Hartley, J. (1996). Popular Reality: Journalism, Modernity, Popular Culture. London: Arnold

  Hedges, I. (1991). Breaking the Frame - Film Language and the Experience of Limits. Bloomington: Indiana University Press.

  Holtzman, S.R. Digital Mantras - The Languages of Abstract and Virtual Worlds. MIT Press, Cambridge, Massachusetts, 1995.

  Isenhour, J.P. The Effects of Context and Order in Film Editing. AV Communications Review, 23 (1). 69-80.

  Jenkins, H. Textual Poachers: Television Fans & Participatory Culture. Routledge, New York, 1992.

  Jordan, T. H. (1975). The Anatomy Of Cinematic Humor. New York: The Revisionist Press.

  Katz, S. D. (1991). Shot by Shot - Visualizing from concept to screen. Stoneham, MA: Michael Wiese Productions in conjunction with Focal Press.

  Kracauer, S. (1960). Theory of Film: The Redemption of Physical Reality. New York: Oxford University Press.


  Kuleshov, L. (1974). Kuleshov on Film - Writing of Lev Kuleshov. Berkeley: University of California Press.

  Manovich, L. The Language of New Media. The MIT Press, Cambridge, Massachusetts, 2001.

  McAleese, R. (1985). Knowledge and Information Mapping and Interactive Video. Aberdeen: University Teaching Centre, University of Aberdeen.

  Metz, C. (1974). Film Language: A Semiotic Of The Cinema. New York: Oxford University Press.

  Metz, C. (1976a). Current Problems of Film Theory: Mitry's L'Esthétique et Psychologie du Cinema. Vol. II. In B. Nichols (Eds.), Movies and Methods (pp. 568 - 578). Berkeley: University of California Press.

  Metz, C. (1976b). On the Notion of Cinematographic Language. In B. Nichols (Eds.), Movies and Methods (pp. 582 - 589). Berkeley: University of California Press.

  Metz, C. (1981). Methodological Propositions for the Analysis of Film. In: Eaton, M. (Ed.) Cinema and Semiotics (Screen Reader 2). London: Society for Education in Film and Television, op. cit., pp. 86-98


  Metz, C. (1982). The Imaginary Signifier - Psychoanalysis and the cinema. Bloomington: Indiana University Press.

  Monaco, J. How to Read a Film: The World of Movies, Media, and Multimedia: Language, History, Theory. Oxford University Press, New York, 2000.

  Neale, S., & Krutnik, F. (1990). Popular Film and Television Comedy. London: Routledge.

  Nichols, B. (editor) (1976). Movies and Methods I. Berkeley;University of California Press.

  Nichols, B. (editor) (1985). Movies and Methods II. Berkeley;University of California Press.

  Oldham, G. First Cut: Conversations with Film Editors. University of California Press, Berkeley, California, 1992.

  Packer, R. and Jordan, K. Multimedia: From Wagner to Virtual Reality. Norton, New York, 2001.


  Peterson, J. (1994). Dreams of Chaos, Vision of Order - Understanding the American Avant-garde Cinema. Detroit: Wayne State University Press.

  Petric, V. Constructivism in Film. Cambridge University Press, Cambridge, 1987.

  Pudovkin, V.I. Film Technique and Film Acting. Bonanza Books, New York, 1949.

  Rabinger, M. Directing - Film Techniques and Aesthetics. Focal Press., Boston, 1989.

  Reisz, K., & Millar, G. The Technique of Film Editing. Focal/Hastings House, New York, 1969.

  Rieser, M. & Zapp, A. (2002). New Screen Media - Cinema/Art/Narrative. London: BFI.

  Rosenblum, R. and Karen, R. When The Shooting Stops, The Cutting Begins - A Film Editor's Story. Da Capo Paperback, New York, 1979.

  Rush, M. (1999). New Media in Late 20th-Century Art. London:Thames & Hudson


  Salomon, G. and Cohen, A.A. Television Formats, Mastery of Mental Skills, and the Acquisition of Knowledge. Journal of Educational Psychology (69). 612 - 619.

  Salt, B. Film Style and Technology: History and Analysis. Starword, London, 1992.

  Spottiswode, R. J. (1955). A Grammar Of The Film - an analysis of film technique. London: Faber & Faber.

  Stam, R., Burgoyne, R. and Flitterman-Lewis, S. New Vocabularies in Film Semiotics: Structuralism, Post-Structuralism, and Beyond. Routledge, London, 1992.

  Weber, K., The Work Practice of Moving Image Indexing. in AAAI-94 Workshop on Indexing and Reuse in Multimedia Systems, (Seattle, Washington, 1994).

  Wells, P. Understanding Animation. Routledge, London, 1998.

  Woods, W. A. (1985). What's in a link: Foundations for Semantic Networks. In R. J. Brachman & H. J. Levesque (Eds.), Readings in Knowledge Representation (pp. 218 - 241). San Mateo, California: Morgan Kaufmann Publishers.


  Wollen, P. Signs and Meaning in the Cinema. Indiana University Press, Indiana, 1972.

  Young, B. M (1990): Television Advertising and Children. Oxford: Clarendon Press

  Zettl, H. Essentials of Applied Media Aesthetics. in Dorai, C. and Venkatesh, S. eds. Media Computing: Computational Media Aesthetics, Kluwer Academic Publishers, Boston, 2001, 11-38.

  Zettl, H. Sight, Sound, Motion: Applied Media Aesthetics. Wadsworth Publishing. Company, Belmont, California, 1999.