2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

PAGAN: Platform for AudiovisualGeneral-purpose ANnotation

David MelhartInstitute of Digital Games

University of MaltaMsida, Malta

david.melhart@um.edu.mt

Antonios LiapisInstitute of Digital Games

University of MaltaMsida, Malta

antonios.liapis@um.edu.mt

Georgios N. YannakakisInstitute of Digital Games

University of MaltaMsida, Malta

georgios.yannakakis@um.edu.mt

Abstract—This paper presents an online platform, namedPAGAN, for crowdsourcing affect annotations. The platformprovides researchers with an easy-access solution for labellingany type of audiovisual content. The tool currently features anannotator interface, which offers three different time-continuous,dimensional annotation tools. PAGAN aims to serve as a freeonline platform for crowdsourcing large affective corpora—required from data-hungry machine learning methods for mod-elling affect—through a publicly available webpage, which is easyto share and use by participants.

Index Terms—Affective computing, human-computer interac-tion, affect annotation, crowdsourcing, video

I. INTRODUCTION

With the advances of machine learning techniques requiringgrowing amounts of data, collecting large volumes of reliableaffect annotations is becoming an increasingly critical chal-lenge in affective computing. Although contemporary toolsfor affect annotation exist, they often require installationor programming knowledge and can often run only undera researcher’s supervision. The PAGAN platform1 aims toaddress issues as such by offering an easily accessible onlineplatform, which can help researchers crowdsourcing and man-aging their audiovisual annotation tasks. PAGAN provides ahighly customisable pipeline for setting up annotation projectswith three different time-continuous annotation tools.

PAGAN focuses on continuous and primarily dimensionalannotation. Dimensional frameworks on simple affective com-ponents such as arousal and valence [1] are preferred for thetask of annotation as they minimise criterion biases of cate-gorical frameworks (e.g. [2]). Although applications based ondimensional theories are unable to capture complex emotionswithout expert interpretation, this simplicity leads to a lowercognitive load and higher face validity during measurement.Unsurprisingly, many contemporary annotation tools [3]–[5]build on a dimensional understanding of emotions. The mainbenefit of continuous annotation tools compared to traditionalmethods (e.g. [6]) is their ability to capture temporal dynamicsof the experience While most contemporary tools [3], [4],[7] rely on a bounded signal to ensure a universal scaleamong raters for the benefit of a wide array of statistical andmachine learning approaches, new ordinal affect annotation

1http://pagan.institutedigitalgames.com

Fig. 1. RankTrace interface in the PAGAN platform. Annotating a conversa-tion with Spike, an agent from the SEMAINE database [9].

techniques emphasise the relative nature of emotions [8]through unbounded labelling [5]. PAGAN includes discrete,absolute scale, and relative unbounded annotation techniquesto cater to a wide variety of research needs.

II. SYSTEM OVERVIEW

PAGAN was designed to require no installation. The plat-form can run in any modern browser and only requires adesktop computer with a conventional keyboard. PAGANconsists of an annotator interface with associated tools andan administration dashboard, that are both described in thissection.

A. Annotator Interface and Tools

PAGAN separates the annotator application and the adminis-tration interface to eliminate distractions during the annotationsessions. Upon navigating to the project link, the annotatoris either greeted with a prompt to upload (or link) a video,or she is sent to the main application with a video alreadyloaded for them. Here, the annotator is welcomed by a shorttext and description of the task, which they can start at theirleisure. The annotator application is controlled by the up anddown keys on the keyboard (Fig. 1). The session is repeated

978-1-7281-3891-6/19/$31.00 ©2019 IEEE

Fig. 2. Project creation screen on the researcher interface.

if the participant annotates less than 25% of the content andit automatically pauses if the annotator navigates away fromthe window but keeps it open. After the session, an optionalmessage and survey link is displayed.

There are tree types of annotation tools implemented inPAGAN. RankTrace is based on the work of Lopes et al.[5] and implemented as an ordinal annotation tool; RankTracedisplays the whole session history (see Fig. 1) which acts asa dynamic reference point during the annotation. GTrace isbased on the work of Cowie et al. [3] and implemented asan interval annotation tool; GTrace provides temporary cuesabout the previous few cursor positions as a form of referencepoint during annotation. BTrace is a new tool based on thework of Yannakakis and Martinez [4] and implemented as adiscrete but time-continuous annotation tool; BTrace measuresbinary change and similarly to RankTrace, displays the wholeannotation history.

B. Administration Dashboard

The administration dashboard is a page dedicated to re-searchers where they can create new (and access ongoing)projects. User data is secured by a user name and encryptedpassword. Researchers can create new projects through ahighly customisable interface, which can be seen in Fig. 2. Theproject tile mainly helps identifying the project on the sum-

mary page. The annotation target is the label used for the y axisin the annotator application (see Fig. 1). The target descriptioncan be used to clarify the annotation task before it starts. Theannotation type features the three aforementioned annotationtools. As a project source, researchers have the option to eitherupload videos, provide YouTube links or task participants withuploading or linking audiovisual content during the annotationprocess. Videos can be loaded both in sequence or randomly,with an option to either end the annotation sessions after aset number of tasks or reshuffle the videos indefinitely. If thevideos are not muted, participants will be reminded to turn ontheir speakers. Optional instructions can be added at the end ofthe annotation procedure to help the integration of the platforminto larger experimental designs. Finally, the system supportsGoogle Forms surveys to be distributed to the participants withan option to auto-fill the randomly generated participant ID.

After creating a project, its progress (how many participantshave completed it) and data (annotation logs and includedvideos) can be accessed under the My Projects page. Here,researchers can find the generated project link as well. Dissem-inating the task to remote participants is done easily throughsharing this link with them. The platform is suitable forsnowball sampling as participants can reshare the project linkto recruit more subjects.

III. CONCLUSION

This paper presented a highly customisable and accessi-ble online platform that democratises and eases the video-annotation task by aiding affective computing researchers andpractitioners to crowdsource such tasks in a rapid manner.The presented framework, PAGAN, offers a free easy-to-useplatform for the collection of large affective corpora. The aimof the project is to enable the wider application of often data-hungry machine learning methods for affect modelling.

REFERENCES

[1] J. A. Russell, “A circumplex model of affect.” Journal of personality andsocial psychology, vol. 39, no. 6, p. 1161, 1980.

[2] P. Ekman, “An argument for basic emotions,” Cognition & emotion, vol. 6,no. 3-4, pp. 169–200, 1992.

[3] R. Cowie, M. Sawey, C. Doherty, J. Jaimovich, C. Fyans, and P. Stapleton,“Gtrace: General trace program compatible with emotionml,” in 2013Humaine Association Conference on Affective Computing and IntelligentInteraction. IEEE, 2013, pp. 709–710.

[4] G. N. Yannakakis and H. P. Martinez, “Grounding truth via ordinalannotation,” in Proceedings of the Int. Conf. on Affective Computing andIntelligent Interaction. IEEE, 2015, pp. 574–580.

[5] P. Lopes, G. N. Yannakakis, and A. Liapis, “RankTrace: Relative andunbounded affect annotation,” in Inte. Conf. on Affective Computing andIntelligent Interaction. IEEE, 2017, pp. 158–163.

[6] J. D. Morris, “Sam: the self-assessment manikin. an efficient cross-cultural measurement of emotional response,” Journal of advertisingresearch, vol. 35, no. 6, pp. 63–69, 1995.

[7] J. M. Girard, “Carma: Software for continuous affect rating and mediaannotation,” Journal of Open Research Software, vol. 2, no. 1, 2014.

[8] G. N. Yannakakis, R. Cowie, and C. Busso, “The ordinal nature ofemotions: An emerging approach,” IEEE Transactions on Affective Com-puting, 2018.

[9] G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, “Thesemaine database: Annotated multimodal records of emotionally coloredconversations between a person and a limited agent,” IEEE Transactionson Affective Computing, vol. 3, no. 1, pp. 5–17, 2012.