Machines Learning Culture

M achine learning, big data, predictive ana-lytics, and deep learning are all com-

monly referenced topics in today’s computingand business worlds. Commercial entities areinterested in such approaches for predictingcustomer churn or preventing fraud, while aca-demics are busy identifying rich challengesrelated to acquiring and cleaning training dataand developing large-scale algorithms. Thepopular press and social media have generatedconsiderable excitement about the potential ofthese innovations to greatly enhance or deci-mate humanity—possibly even both at thesame time.1

In May 2014, a number of celebrated scien-tists, including Stephen Hawking, authored acolumn in a UK newspaper, The Independent,that struck a cautionary note in discussing thepossible threat of fully realized artificial intelli-gence.2 Using the (somewhat suspect) narrativeplot of the Hollywood movie Transcendence as aframing device, the authors highlighted thechallenges of reaping the benefits of advancesin AI and advocated the need to study possibleincalculable risks. Subsequent commentary onthe newspaper website and elsewhere debatedthe merits of the overall argument, with manyweighing in using references to other specula-tivemovies and science-fiction literature.

The article’s movie-framing gambit clearlysucceeded in captivating the attention of con-siderable sections of the online public, motivat-ing diverse discourse about a complex topic.This leads me to consider what other forms ofcultural practices and production might simi-larly engage audiences in contemplating thepromise, beauty, menace, and power of big-data-stylemachine learning?

Some recent interdisciplinary explorations,integrating contributions from computer sci-ence, math, the digital humanities, and thearts, point to the utilitarian and expressivecapabilities of machine-learning approaches increating work with diverse appeal. These initia-tives include research within the relatively tra-

ditional domain of historical art analysis, agrowing collection of body-tracking work usingmachine learning quietly in the background,and a variety of provocative art installationsthat place algorithmic computing front andcenter. While these projects tackle their subjectat varying levels of scale and depth, and in dif-ferent contexts, each contributes to buildingthe public discourse about the impact, role, andreach ofmachine learning in our lives.

Visual Art AnalysisThe role of an art historian includes the provi-sion of a context within which a reader orviewer can better appreciate the significance ofa considered work. Recent research from theDigital Humanities Research Laboratory atRutgers University points to the potential ofmachine-learning techniques to assist (but notyet replace) historians in efficiently identifyingevidence of possible artistic influence acrossgroups of paintings.3 While such inferences ingeneral are naturally subjective, insights gainedusing multiple classification techniques cancertainly help support or challenge particulararguments.

In “Toward Automated Discovery of ArtisticInfluence,” the authors carefully note that theirapproach is not “asserting truths but insteadsuggesting a possible path towards…measuringinfluence.”3 Using a dataset of 1,710 images ofartworks (primarily paintings) by 66 artists cre-ated over approximately 550 years, the authorsexperiment with several discriminative andgenerative methodologies, before exploringlow-, intermediate-, and semantic-level fea-tures. In their findings (see Figure 1), they indi-cate that their approach highlights possibleevidence of artistic influence between paintingswhich, to their knowledge, have not been dis-cussed together before, including FredericBazille’s L’atelier de Bazille (1870) and NormanRockwell’s Shuffleton’s Barbershop (1950).

While acknowledging that their work is justbeginning to scratch the surface of automated or

Aisling KelliherCarnegie Mellon

University

Aisling KelliherCarnegie Mellon UniversityArtfulMedia

1070-986X/15/$31.00!c 2015 IEEE Published by the IEEE Computer Society18

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

semi-automated discovery of artistic influence,such endeavors point to the value of integratingcomputer science and digital humanities as arich application space for evaluating andstretching machine-learning methods and per-formance and as a way for testing implicit orsubjective judgments.

Body TrackingOff-the-shelf and customized machine-learningtechniques are increasingly part of the toolsetof electronic media artists interested in exploit-ing the potential of this computationalapproach. In the Augmented Hand Series proj-ect, collaborators Golan Levin, Chris Sugrue,and Kyle McDonald have created a physicalinstallation containing “a real-time interactivesoftware system that presents playful, dream-like, and uncanny transformations of itsvisitors’ hands” (see www.flong.com/projects/augmented-hand-series). Their system usesstandard vision and machine-learning techni-ques (including the Leap motion library) toanalyze the posture of participants who inserttheir hand into a box, with the resulting trans-formed image reimagining the displayed handwith additional fingers or autonomously mov-ing digits (see Figure 2).

The creators of the Augmented Hand Seriesnote the challenges that the public exhibition oftheir work brings up, including dealing with avariety of hand sizes, skin colors, skin ages, orna-

mentation, and general divergent finger features,all of which require additional and ongoingresearch as they developnew forms of transforma-tion that delight and engage expanded audiences.

Provocative Art InstallationsMachine learning is brought to the fore inworks that aim to directly translate and

Figure 1. Comparison of the Bazille and Rockwell paintings, where yellow annotation indicates object similarity, red demarks

compositional similarity, and blue denotes structural similarity. (Source: Ahmed Elgammal; used with permission.)

Figure 2. A young child experiencing the Augment Hand Series installation.

The system uses standard vision andmachine-learning techniques to analyze

the posture of participants who insert their hand into a box, with the resulting

transformed image reimagining the displayed hand. (Source: Golan Levin;

used with permission.)

April–Ju

ne2015

19

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

________________

represent unfolding algorithmic processes in aformat that is legible to non-expert viewers.

RepresentingMachine VisionBenjamin Grosser’s work aims to inform view-ers about the underlying process by which a

machine vision system differs from ours interms of how it might “watch a movie.” In hisproject, Computers Watching Movies, (seehttp://bengrosser.com/projects/computers-watching-movies), computer vision techniquesand AI processing are integrated in melding acomputational approach that instills a degreeof agency to the viewing machine. The atten-tion and disinterest of the algorithm is drivenby the prominent features detected in theviewedmovie frame, which in turn is demarkedby an ever-shifting line trace drawn to matchthe dynamic computer gaze. The algorithm isdirected to watch scenes from contemporaryfast-paced movies such as Inception (see Figure3a) and The Matrix, in addition to more charac-ter-driven fare such asAnnie Hall and Taxi Driver(see Figure 3b).

The resulting analysis is presented as a tem-poral animation accompanied by the real-timeaudio from the attended scene. While theresulting animations make for gently hypnoticviewing, the relative lack of cohesion betweenthe audiovisual interpretation and the well-understood-by-humans narrative points to avery different model of message or mediuminterpretation. The salient meaning derived byhumans—based on years of exposure to moviesand indeed, general life experience—is at oddswith the limited training of the algorithm on aconstrained dataset. Building on Stuart Hall’stheoretical model of encoding and decodingcommunication, what is encountered here is aradically different contextual decoding frame-work,where the translation of significance, orig-inality, or import could initially be understood asoppositional, before potentially moving, overtime, toward a negotiated or even hegemonic

(a) (b)

Figure 3. Still frames from the ComputersWatchingMovies project. The algorithmwatches scenes from (a) the fast-pacedmovie

Inception, and (b) themore character-drivenmovie, Taxi Driver. (Source: Benjamin Grosser, http://bengrosser.com; used with

permission.)

8.1

7.2

2.6

Turing normalizing machine

9.7

Figure 4. Promotional image for the Turing Normalizing Machine, a

collaborative experiment that aims to decode what society deems “normal.”

(Source: Mushon Zer-Aviv; used with permission.)

Artful Media

20

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

____________

_____________________________

position.4 Perhaps Grosser’s work could point(even somewhat tongue-in-cheek) toward aworldwhere new forms of automated critique and anal-ysis help us better consider that which is novel,different, and (eventually)meaningful?

Exploring Social NormsMachine learning is also placed front and centerin works such as the Turing NormalizingMachine, a collaborative experiment by Yona-tan Ben Dimhon and Mushon Zer-Aviv. Here,the artists posit their approach as “experimentalresearch in machine learning that identifiesand analyzes the concept of social normalcy”(see http://mushon.com/tnm).

Inspired by, and in tribute to, Alan Turing’sdocumented life experiences and contributionsto the fields of computing and artificial intelli-gence, the installation presents participantswith four video images of different peopleengaged in the act of pointing. The participantsare then instructed to point at the image depictingthe person or people they deem “most normal.”Their selection is analyzed and its visual featuresare added to “an algorithmically constructedimage of normalcy” (see Figure 4). In addition, avideo recording of the participant making theirselection is captured and added to the system forlater use. Over time, the system develops a morerefined model of so-called “normal presenting”appearance, bringing the artists closer to theirgoal of decoding “the mystery of what societydeems ‘normal.’” As commentary on the abjectprosecution of Alan Turing by a prejudicial soci-ety, the piece both highlights and questions thepower of human-aided machine-learning systemsin classifying socially determined norms aboutwhat is palatable or acceptable.

Visualizing ContentThe artwork and designs of other practitionersgive pause in consideration of how their workcould enhance, but yet still resist,machine-learn-ing practices. For example, designer Stefanie Pos-avec’s exquisite visualizations of features—suchas the underlying textual structures, sentencerhythms, and overarching themes of the worksof several noted 20th century authors—were cre-ated by undertaking painstaking by-hand analy-sis of the primary texts.

In a process she describes as “knit(ing) togetherinformation visualizations,” the laborious processof analyzing the text by hand—as opposed to“pressing ‘enter’ on a keyboard”—adds to the sig-nificance of the work (see www.stefanieposavec.

co.uk/-everything-in-between/#/writing-without-words). With her declared focus on the meaningof the analyzed content as derived from carefulreading and re-reading, her analog approach tothe poetic rendering of extracted features pointsto a compelling motivation to keep humans inthe machine-learning loop, whether as a teacher,an apprentice, a client, a subroutine, or a highlysubjective and critical interpreter.

A s we continue to encounter a world ofrecommendations and predictions based

on somewhat black-box computational processes,

Figure 5. Literary Organism, by Stefanie Posavec. This work analyzes textual

features to visualizemeaning. (Source: Stefanie Posavec; used with permission.)

April–Ju

ne2015

21

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

____

________________________________

perhaps the human injection of the unfamiliar,the surprising, the uncanny, or even the devi-ant, will help ensure a refreshing critique oreven rejection of the aesthetically prominentin favor of a different, more subtle flavor ofcreative innovation. MM

References

1. G. Marcus. “WhyWe Should Think About the

Threat of Artificial Intelligence,” The New Yorker, 24

Oct. 2013; www.newyorker.com/tech/elements/

why-we-should-think-about-the-threat-of-artifi-

cial-intelligence.

2. S. Hawking et al., “Transcendence Looks at the

Implications of Artificial Intelligence—But Are We

Taking AI Seriously Enough?” The Independent, 1

May 2014; www.independent.co.uk/news/sci-

ence/stephen-hawking-transcendence-looks-at-

the-implications-of-artificial-intelligence–but-are-

we-taking-ai-seriously-enough-9313474.html.

3. B. Saleh et al., “Toward Automated Discovery of

Artistic Influence,”Multimedia Tools and Applica-

tions, Aug. 2014; doi: 10.1007/s10042.

4. S. Hall, “Encoding and Decoding in the Television

Discourse,” Culture, Media, Language: Working

Papers in Cultural Studies, Centre for Contemporary

Cultural Studies, 1973, pp. 128–138.

Aisling Kelliher is an associate professor in the

School of Design at Carnegie Mellon University.

Contact her at aislingk@andrew.cmu.edu.

Artful Media

22

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

_________________________

________

_______________

___________________________

____________________

_________

___________________________

__________________________

____________________________

________________