Transformation through Provocation?Designing a ‘Bot of Conviction’ to Challenge Conceptions and Evoke Critical

Reflection

Maria RoussouNational and Kapodistrian University

of AthensAthens, Greece

mroussou@di.uoa.gr

Sara PerryUniversity of York

York, UKsara.perry@york.ac.uk

Akrivi KatiforiAthena Research & Innovation Center

Athens, Greecevivi@di.uoa.gr

Stavros VassosHelvia Technologies

Athens, Greecestavros@helvia.io

Angeliki TzouganatouUniversity of HamburgHamburg, Germany

angeliki.tzouganatou@uni-hamburg.de

Sierra McKinneyUniversity of York

York, UKslm589@york.ac.uk

ABSTRACTCan a chatbot enable us to change our conceptions, to be crit-ically reflective? To what extent can interaction with a tech-nologically “minimal” medium such as a chatbot evoke emo-tional engagement in ways that can challenge us to act on theworld? In this paper, we discuss the design of a provocativebot, a “bot of conviction”, aimed at triggering conversationson complex topics (e.g. death, wealth distribution, genderequality, privacy) and, ultimately, soliciting specific actionsfrom the user it converses with. We instantiate our designwith a use case in the cultural sector, specifically a Neolithicarchaeological site that acts as a stage of conversation onsuch hard themes. Our larger contributions include an in-teraction framework for bots of conviction, insights gainedfrom an iterative process of participatory design and evalua-tion, and a vision for bot interaction mechanisms that canapply to the HCI community more widely.

CCS CONCEPTS• Human-centered computing → Interaction design the-ory, concepts and paradigms; Interaction techniques.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACMmust be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from permissions@acm.org.CHI 2019, May 4–9, 2019, Glasgow, Scotland UK© 2019 Association for Computing Machinery.ACM ISBN 978-1-4503-5970-2/19/05. . . $15.00https://doi.org/10.1145/3290605.3300857

KEYWORDSChatbots; conversational agents; UX design; provocative in-teraction; emotional engagement; cultural informaticsACM Reference Format:Maria Roussou, Sara Perry, Akrivi Katifori, Stavros Vassos, An-geliki Tzouganatou, and Sierra McKinney. 2019. Transformationthrough Provocation?: Designing a ‘Bot of Conviction’ to Chal-lenge Conceptions and Evoke Critical Reflection. In CHI Conferenceon Human Factors in Computing Systems Proceedings (CHI 2019),May 4–9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA,13 pages. https://doi.org/10.1145/3290605.3300857

1 INTRODUCTIONThe study and design of concepts, metaphors, practices, andevaluation methods in User Experience (UX) has been thesteady endeavor of researchers and practitioners working inthe field of human computer interaction (HCI) for a numberof years now. An increasing emphasis in UX is given tothe affective dimension, for example the design of emotive,hedonic [31], enchanting [29], empathic [54] or criticallyreflective [39] [7] [3] [26] approaches to interaction betweenhumans and the digital world.Within this landscape, we havewitnessed a surge of different interactive systems in variousfields (cultural heritage, tourism, education, e-commerce,etc.) that rely on detecting the human user’s emotional stateand responding to it appropriately.

This ‘turn’ to affect [52] in the design of experiences, inter-faces, and interaction methods has, however, been primarilymanifested in systems that attempt to capture users’ emo-tional states and offer, in return, a relevant response. Rarelyis the user’s digitally mediated emotional engagement withthe content regarded as an opportunity to trigger a deeperconnection, to critically reflect on the issues at stake, to chal-lenge and provoke a call to action.

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 1

Provoking this kind of “conversation” wherein the humanparticipant can be challenged into thinking about what theirprinciples or assumptions actually mean and, subsequently,act on them to transform their experience, is at the core of theworkwe propose in this paper. Based on an affective practicesmodel of emotional engagement [53] and inspired by bothGraham’s [13] call for digital media that are able “to move us,to inspire us, to challenge us,” and his reference to Sample’sconcept of “bots of conviction” [36], we engaged in designinga conversational agent (CA), or chatbot. Its aim is to evoke itsuser’s emotional engagement with complex topics (e.g. death,wealth distribution, gender equality, privacy) and, ultimately,solicit specific actions from the user it converses with.

We chose to explore the design of a chatbot because it is a“minimal” digital medium, it is direct and simple to use, and itis playful. But how can we design conversational interactionwith a chatbot in ways that can trigger critical reflection? Towhat extent can interaction with such a technologically “min-imal” medium bring out deeper emotions that can challengeus to act on the world?In this paper, we introduce an interactional pattern that,

we argue, can ignite a dialogue between a participant anda bot, aiming ultimately to transform the participant’s con-ceptions. We start by defining key concepts related to ourgoals of emotional engagement, provocation and transforma-tion. We then review the variety of chatbots used today, withparticular emphasis on chatbots used in the cultural sector,as this is where our use case is situated. Next, we describethe iterative process of designing a Bot of Conviction (BoC),which follows a carefully planned out and executed, throughformative evaluation, procedure of content and interactiondesign and development. Section 5 demonstrates how weapply our pattern to the design of a chatbot for a specific ar-chaeological site. Finally, the paper concludes by discussingour pattern, its limitations and its potential to fulfill the goalof igniting users’ transformation through a call for action.

2 BACKGROUNDChatbots and the Post-app InternetThe literature on conversational agents, intelligent virtualhumans, virtual assistants, and chatbots is extensive. Whilstnot exactly the same [51], these terms are often used in-terchangeably to denote systems that engage the user, to agreater or lesser degree, in natural language-like conversa-tion (spoken or written) with a digital entity.Chatbots have been touted as advantageous tools that

can facilitate communication, provide easier access to in-formation, and combat digital divides [9]. They offer novel,immediate engagement mechanisms and, in light of the pop-ularity of texting, they can attract a younger demographicin multimodal ways [49]. In addition, chatbots can operate

on both a browser and a mobile phone, offering a solutionto the challenge of app installation and overload [9].Chatbots serve a broad range of purposes [32], with the

most common application being that of a first-level helpdesk or service chatbot that can recommend responses tolow-level customer queries. These chatbots lower the thresh-old for people to ask for information, work well on simpleissues, and provide a more amiable and personable style ofinformation delivery. As customer service chatbots becomecommonplace, the field is now turning to advancing thecreation of agents that are able to build relationships withtheir human conversational partners [28] as well as virtualhumans that can converse with the user in more emotive,persuasive, and provocative ways [38] [40]. At the same time,conceptual and ethical issues are informing the design ofguidelines for bots [21] [37].Despite the aforementioned attempts, in the majority of

conversational agents, the typical form of interaction is anindependent single-turn exchange: the user asks, the chatbotresponds, and this usually completes the interaction for theparticular question/topic.

Chatbots in the Cultural SectorConversational interfaces are increasingly espoused by cul-tural organizations within their digital strategies as meansto attract new audiences and extend the museums’ physi-cal location. They are regularly proclaimed to offer novelengagement mechanisms that can empower visitors of muse-ums and broaden the ways that cultural content is perceived.Many current cultural heritage chatbot initiatives operatewithin a site’s physical space, allowing for varying levelsof interactivity. The chatbots’ most usual in situ purpose isserving as exhibition guides [20] [50] [6] and helping visitorsin organizing their visit [15]. These bots resemble customerservice bots, as their primary aim is to offer information tothe visitor.

In more sophisticated examples, visitors input a keyword,color or even an emotion, and the chatbot will respond witha selection of related artworks [1]; or interact with embodiedvirtual agents in the informal education space (e.g., Adaand Grace [46], Max [19], Coach Mike [22], Alan Turing’sAvatar [12]), either by spoken natural language or via typedtext. Some embed gamification elements into their touringfunctionality [19] [48], challenging users with exploratoryclues or quizzes that manifest in rewards, including virtualcurrency that has cash value in museum gift shops.However, despite these examples, the use of conversa-

tional agents by museums and the heritage sector is stillquite limited. Most are purely info-delivery oriented andobject- or exhibit- centered, providing little opportunity formeaningful interactivity, creative expression, or critical en-gagement. In response to these limitations, we seek to extend

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 2

the “traditional” canon of the museum/heritage bot into achallenging, provocative engine of social commentary andself-reflection.

3 DEFINING EMOTIONAL ENGAGEMENT,PROVOCATION, AND TRANSFORMATION

Contemporary definitions of emotion and affect (e.g. see [52]and [53]) increasingly aim to depart from the psychobio-logical “basic emotions” approaches, which reduce affect tosimplistic innate human universals and do not account forthe multiplicity of factors that mix in any given individual’saffective practices. Rather, in recent and more complex con-ceptualizations of the term, emotion is framed as “embodiedmeaning-making” [52, p.4], and focus is put on the actionsthat are generated through such embodied work (actionsthat may be small or large, personally-oriented or externally-oriented, visible or invisible, etc.). Recognizing that emotionhas action embedded into it allows us to attend to the affec-tive practices that characterize meaning-making–the actionsthat feed into and flow out from it. Therefore, rather thantry to crudely measure emotion as biological response, weturn our attention instead to the acts (or lack thereof) thatare generated through people’s practices with our Bot ofConviction.

The importance of such a flexible and act-centered under-standing of emotion cannot be understated. It permits us tooperate in cross-cultural contexts (as our concern with ana-lyzing resulting actions means that we do not need to relyon typical English-language emotion descriptors to defineaffective experiences) and to embrace the true complexityof emotive experiences. It also appreciates the intentionalityand control–but also the historical motivations and personalrelationships–that can be at the core of such experiences. Inline with this conception of emotion, we look for repetitions,apparent inconsistencies and unique occurrences in actions(e.g. spoken or written words, non-discursive oral expres-sions, bodily movements and gestural reactions, interactionswith human and non-human things, other proxemics, draw-ings or other visual inscriptions, etc.) that emerge in people’ssocial practices. In terms of our BoC, this means emotionalengagement is demonstrated via interaction with the botitself and is inherent in the very act of chatting to it. Ratherthan designing the bot to trigger simplistic “basic emotions”,we create conditions inside the chats with the purpose ofsoliciting specific intended actions from participants.As we see it, to respond at all to the bot is to affectively

engage with it (as a user could easily just walk away). Suchbasic response actions suggest the efficacy of the bot’s con-ditions in provoking a reply. Provocation, here, is definedin simple terms: acting on others to elicit a particular recip-rocal action. At the most superficial level, the bot acts onthe user, engaging them sufficiently to complete a full chat.

Preferably, however, this provocation works more deeply,evidenced through analysis of the types of inputs generatedby users. Here, deeper provocation entails users reconsider-ing their points of view, demonstrating forms of consciousreflection or alternative perspective-taking in their chats.Moreover, at its deepest level, as we define it, provocationleads to transformation: users take action beyond the chatitself, for instance telling others about their reflections, orintegrating ideas generated through engagement with thebot into their own everyday meaning-making practices. Heretransformation is loosely aligned with Hennes’ [14, p.114]concern that “The difference between the activity of the be-ginning and that of the end is a kind of transformationalgrowth that affects experience in the future”. In this way, ourdefinition goes further than some in the heritage sector [43,p.104] who see transformation as “simply instances when vis-itors’ sense of self and community [a]re destabilised”. Rather,we are interested in effecting genuine change in individualswhich is evidenced, following Soren’s model of transforma-tional museum experiences [44, p.248], in behaviours whichare “more inclusive, discriminating, emotionally capable ofchange, and reflective”.In creating provocation, it is necessary to consider the

ethical implications of the work on users’ wellbeing. How-ever, drawing inspiration from Katrikh [17] and Gargett [11],it is our position that, in order to develop a transformativeexperience as outlined above, efforts should not lie in mini-mizing discomfort but rather in generating the opportunityto “promote dialogue, process emotion, and ultimately toallow visitors to reach a place of equilibrium” [17, p.7].

4 DESIGNING BOTS OF CONVICTIONWe have turned to the concept of Bots of Conviction to ex-plore the potential for more open conversational agents thatfocus on asking (not necessarily answering) questions, andprovoking critical thought. In particular, our motivation liesin exploring “hard” themes that are emotive and controver-sial in nature, such as life and death, power, wealth, socialstructure and hierarchy (or lack thereof), gender equality,etc. Critically, these topics are relevant across time and space,meaning they shaped people’s lives thousands of years agoin myriad ways and they continue to evolve diachronically,remaining relevant to humans today. Cultural heritage sitsat the center of debates about identity, politics, socialityand economics, regularly appropriated by interested par-ties in the present to justify past actions and to lay claimsto the future. Not often, however, do heritage sites fosterenvironments where such debates are purposefully and con-structively facilitated, such that the resulting dialogue leadsto positive social change [11] [33]. The chatbot presentswhat is arguably the perfect opportunity to experiment withdiscussion-based models of affective engagement. This, then,

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 3

is the “space” our work aspires to occupy, the nexus being toenable genuinely critical reflection, respect, care, and ethicsin dialogue.Seeking inspiration, we looked towards Sample’s defini-

tion of bots of conviction [36]. Otherwise known as “protestbots”, these computer programs work to reveal “the injusticeand inequality of the world and imagin[e] alternatives”; theyask questions about “how, when, who and why”; and theyare typified by five key traits: topicality, uncanniness, accu-mulation, oppositionality and groundedness in data. Unlikethe “typical” BoC, however, which is usually Twitter-based,generative and broadcast oriented (in the sense that it isnot intended to foster a two-way conversational flow), wesought to develop something more amenable to the usualcultural heritage context. Indeed, by Sample’s logic, BoCs arecompletely ‘automatic’ in nature and “do not offer solutions.Instead they create messy moments...” [36]. Yet the museumssector is bound by ethical codes which demand a basic levelof accountability to and responsibility for their audiences.The evidence also indicates that people may purposefullyvisit museums to change their minds, and that such changederives from more than one-way information delivery [33].Moreover, this is a sector wherein practitioners are oftenunderfunded, understaffed, with variable digital expertiseand sometimes little capacity to maintain or manage the fall-out of Artificial Intelligence or uncontrolled generativity. So,while we borrow elements from Sample’s original definitionof the BoC (specifically the concepts of uncanniness andoppositionality), we intentionally modify it to account forthe cultural heritage context and its associated duties of care.

The ContextThis work is situated within a larger project set to explorethe potential emotive connections of visitors to museumsand archaeological sites, and how digital tools can enhancethese sites’ relevance to people’s lives today [16] [35]. Thearchaeological site of Çatalhöyük, a 9000-year-old Neolithicsettlement in Turkey, has been chosen as an ideal use casefor applying the Bot of Conviction.More than 1000 specialists from around the world have

been excavating Çatalhöyük for 60 years, yet only a smallfraction of the settlement (7%) has been unearthed. Sinceits inscription as an UNESCO World Heritage site in 2012,there has been an increase in visitor numbers despite itsremote location in the center of Turkey. What visitors en-counter, however, at the site is essentially an excavation,where features are difficult to see and significance is hard tounderstand or relate to. The interpretation of the archaeolog-ical record remains limited on-site, especially if the audiencelacks archaeological literacy. Nevertheless, interest in thesite is large; nearly 10,000 Facebook users, most of whomwill

never visit the actual physical site, follow the Çatalhöyükexcavation research project.What makes Çatalhöyük unique and relevant to our ap-

plication is that, according to evidence, it was occupied byup to 8000 people at once without obvious hierarchy (i.e.,egalitarian socio-economic organization). No houses withdistinctive features (belonging to royalty or religious hier-archy, for example) have yet been found. There is also noevidence of social distinction based on gender, with menand women seeming to have equal social status. Residentsrepeatedly built and rebuilt their homes on the same spot,creating a mound of more than 21 meters high over 1000years. Exquisite sculptural art and wall paintings, street-lessneighborhoods, and burials of the dead beneath floors ofhomes are further reasons to choose Çatalhöyük as the stageto explore digital forms of interaction with topics that canprovoke the people of today.

The first step towards the realization of our BoC was thedesign and development of a “traditional” infobot, based onkey themes and topics underlying the cultural site of inter-est. This bot would serve as a baseline for understandingthe added value of the BoC and consisted of a lengthy de-sign process. It entailed: i) selecting and curating contentto construct the chatbot’s knowledge base; ii) designing thebot’s form and interaction mechanisms, and programmingits level of “pickiness” when responding to user input; and iii)designing the conversational aspects of the bot in a way thatcould encourage the kind of action we sought. We presenteach of these steps in the design process below, elaboratingon (iii).

Curating the ContentThe content (themes and topics) of conversation is of utmostimportance in a chatbot that aspires to provoke its audience.We followed an inclusive approach to content selection andcuration that sought the involvement of both content domainexperts and end-users. We recruited domain experts andheld live chat sessions with end-users to first create basiccontent for the bot. From there we elaborated the bot withmore complex reflective and emotional components, to tiethe topics to the deeper underlying message or overarchingthemes.

User-led content curation. As our first use case, presented indetail in Section 5, pertained to the aforementioned StoneAge archaeological site of Çatalhöyük, and as we aimed tocreate a user-centered experience, it was critical to begin byreviewing the Facebook page of the Çatalhöyük ResearchProject, from its profile creation in 2010 to April 2017 (251posts). By researching Facebook followers’ reactions, com-ments and interests, we had the opportunity to create rel-evant content tailored to the ‘needs’ of the user. Thus, we

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 4

researched the types of posts and the types of commentson posts. This thematic analysis focused on grouping andselecting the topics that people seemed to be more engagedwith, as evidenced by their posing questions below postsor by liking, commenting or sharing [49]. In other words,the selection process focused on relevance. People mostlyoffered comments regarding the following topics: burials,wall paintings, archaeological process, the site’s landscapeand importance, chronology, plastering and figurines. Thisfirst phase of selection led to an early design of the chatbotin terms of the topics it would be conversing about.

Live chat sessions. To further develop the bot’s content, thisearly selection of content was augmented, refined and testedthrough a series of live chat sessions with the public. Thesessions were held on the topics that were identified by thethematic analysis and then validated by domain experts, al-lowing us to construct the bot’s factual database. Specifically,five live chat sessions were conducted on the site’s Facebookpage, between June and October of 2017. Domain expertson each topic were recruited and assigned to a session, withsessions covering the topics of i) burials, ii) coprolites andlatrines (poop and toilets!), iii) the archaeological processand excavations, iv) wall paintings, and v) wall reliefs andplastering. The sessions were advertised through the site’ssocial media channels as well as disseminated to the pub-lic via marketing-like posts through the contact lists of ourproject team.Each session began with a post at the appointed starting

time announcing its topic. Followers were then asked to posetheir question below the post or to send a private message.An average number of 17 users connected actively to thelive events across all sessions. Although it was stated in theadvertisements that the live chat sessions would last onehour, users continued to post questions up to ten hours aftersome of the sessions.

Designing InteractionThis next step involved integrating the content from the the-matic analysis and the live chat sessions into the bot, andembellishing it with richer media and a more evocative form.This process was, at its core, a design process, both in termsof the design of conversation and the creation of the inter-action mechanisms and visual elements that make up thechatbot’s “character”. It revealed a set of design challengesin different aspects of the bot and highlighted the need forthe identification of guidelines and best practices in the field.Design decisions that had to be made included:

Language style. Informality is a defining trait of chatbotpersonality but how chatty, witty or funny should a bot be,especially if it is intended as a Bot of Conviction, consideringthe diverse public it targets?

Casual vs non-casual content. Should the bot contentbe mostly non-casual, i.e. mostly information that expertshave prepared about the topic at hand? Or, more casual,including responses to everyday questions, e.g. about theweather or the user’s mood that day?

Canned reply controls. What is the right balance be-tween buttons and quick replies (“canned reply controls”)and free text input? In particular, should users click on pre-defined answers on buttons to make selections or be able toask free-form questions?

Alternative responses.Howmany alternative responsesto the same questions are sufficient, to allow the bot to replyin slightly different ways, so that responses would not seemformulaic should the user repeat the chat?

Picky vs non-picky. What is the pickiness of the chat-bot’s responses?When the chatbot is “picky”, it only matchesa response when there is high confidence on similarity, soit answers, “I do not know” in most cases when it gets aquestion it cannot recognize. A non-picky bot matches aresponse when there is also low confidence on similarity, sowhen asked something it will return an answer even thoughthe matching score is very low.

Use of multimedia.What is the best use of images, linksand emojis within the bot?

Making the bot personable. How can we develop a fewkey elements that will make the bot personable and theexperience personalized and natural for the user? E.g., thebot addressing the users with their (Facebook) names whenchatting.Following the study described below, the infobot param-

eter configuration was fixed to the most effective variant(witty / casual / with buttons for user replies but free text aswell / alternative bot responses / non-picky / use of images /personal with referring to user’s first name).

Formative evaluation. To test our design decisions as wellas the early prototype of the bot, we conducted a studywith 27 participants (14 men, 13 women, aged between 21and 57), located in different countries and with differentbackgrounds (in terms of expertise in relation to the content).Specifically, we recruited researchers from the ÇatalhöyükResearch Project team (2 users: 1 man, 1 woman), followersof the Çatalhöyük Research Project Facebook page (5 users:2 men, 3 women), and people who had no prior knowledgeabout the site and hence were completely unfamiliar with itsstories and significance (20 users: 11 men, 9 women). Someparticipants were involved in the live sessions but, otherwise,had not interacted with the chatbot before.

The evaluation sessions were carried out with each personseparately, either face-to-face or remotely. After completinga consent form, participants were instructed to use Messen-ger to interact with the chatbot in order to learn more about

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 5

the UNESCO World Heritage archaeological site of Çatal-höyük. They were advised to converse freely with the botfor approximately 10 minutes and then asked to undertakespecific tasks, namely respond to questions that the bot had(e.g. “Where did the people of Çatalhöyük bury their dead?”)and did not have (e.g. “Did the people of Çatalhöyük play?”)answers to. The session was followed by a semi-structuredinterview (conducted via Skype in the cases of remote partic-ipants) and an online questionnaire. All interactions betweenbot and users were logged and timestamped.The results of this formative study are briefly outlined

here, in relation to our objectives. Firstly, with regards tousers’ engagement with the bot, participants reported tohave found it interesting and spent an average time of 16.08minutes chatting with it. The majority of users consideredthe images very helpful and enjoyed the anthropomorphicpersona of the chatbot. This is consistent with the findingsof other researchers who note that playful interactions are akey aspect of the adoption of CAs [23] [24] and that peoplereact socially to virtual characters–even if they know thatthey are conversing with a machine [51].

However, as soon as the bot was faced with questions thatit could not understand and thus could not reply to appro-priately, users reported losing interest. It seems that usersexpect what Cassell refers to as “interactional intelligence”[4], the “social smarts” that would enable engagement [5][24]. Instead, the bot offered mostly “propositional intelli-gence”, i.e. informational content upon request, like convers-ing with a knowledge domain expert. However, chatting isgenerally associated not just with information exchange, butalso the exchange of perspectives and opinions. Therefore,the next natural step was to equip the bot with the possibil-ity to hold a meaningful dialogue with its users, challengingthem to approach the presented topic through a completelynew perspective.

Designing for ProvocationThe results from the formative evaluation of the chatbot’scontent and form informed our next step which was to exper-iment with the insertion of patterns of provocation that playwith the idea of the Bot of Conviction. To create our BoCwe developed a conversational pattern that enables the botto initiate a kind of “Socratic dialogue”, where the chatbotembarks on a soft interrogation, asking questions to find outmore about the other person’s beliefs and ideas, while stillmaintaining control over the structure and direction of theconversation.

This pattern resembles a figure-8 (Figure 1). It entails thebot making a declaration designed to commit the user toa point of view that they may or may not agree with. Itbegins with the bot either asking a question or making abold statement. This prompts the user to respond, either

Figure 1: A figure-8 design pattern for a Bot of Conviction.

positively or negatively or neither, and continue further intothe conversation. In other words, a dialogue between the userand the bot plays out based on one of three types of response(yes, no, ambiguous). After a few exchanges, the users willbe questioned about their response to the topic, the centerof the formation, before entering the second section, whichconcludes with a summarizing statement. This statementis one of intent/conviction by the bot, meant to affirm andtransform the point of view of the user, thereafter pushingthem back out into the traditional/standard experience.The structure engages the user by reversing the roles of

the traditional infobot, with the bot asking questions firstthus provoking users to generate the answers, all while main-taining a guided and controlled exchange. According to thepattern, the user’s answers place them on distinct paths. Thebot’s responses are designed to be sensible for a variety ofuser responses, and consistently incorporate questions toprovoke a user response (Figure 2).

The pattern in a nutshell:

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 6

(1) Bot makes a declarative value judgment - a provoca-tion.

(2) User responds positively (variations of yes), negatively(variations of no), or ambiguously (everything else).

(3) Exchange of ideas: 2-3 interactions based on whetherthe user is categorized as positive, negative or am-biguous. In all cases, the user’s response to the bot’squestion should be one of these three categories.

(4) Assessment points. Partway through the dialogue, thebot tests the user’s conviction.

(5) Final statement of intent/conviction - culmination ofthe provocation.

Figure 2: The BoC design pattern “algorithm”.

Entryway into the BoC. The BoC pattern is blended intothe more traditional informational bot rather than being aseparate program. Users enter into the more challenging/self-reflective dialogue of the BoC via one of the following theme-oriented means:

• In relation to specific trigger words linked to themes(e.g., burial, goddess). In other words, the user enters

one of the words and the conversation based on thepattern is triggered.

• After a number of interactions from the user on aparticular thematic topic (where interaction is a singleinstance of user-bot exchange) or typing/selecting abutton with the word “Intrigue me’.

5 THE DEVELOPMENT OF CHATÇATIn the previous sections we outlined why we consider provo-cation to be an important underlying approach to design-ing interaction with conversational agents. We went on topresent the iterative design of such provocative agents, orBots of Conviction, which encompasses a pattern of provo-cation that culminates in a statement of conviction. In thissection we illustrate our design method through an instanti-ation, a Facebook Messenger-based conversational interfacefor the archaeological site of Çatalhöyük, named ChatÇat(Figure 3), that aims to inform about the site and, more im-portantly, to compel critical reflection about the past andaction in the present amongst its users.The rationale behind designing, creating and developing

a chatbot for the particular site of Çatalhöyük is twofold:to address the challenges that the site faces by offering adigital experience of it; and to leverage the many threadsof conversation that the site has to offer and that can bedeveloped around it.

Applying the Design PatternFrom a multitude of topics, we selected four themes to applythe BoC pattern to. In keeping with our pattern philosophy,each “episode” begins with an opening question conceivedto provoke the user’s reaction:

• Death: Would you bury someone you care about underyour bed? Or: Surely, you have people buried underyour floors?

• Wealth: Do you live in a community where there are afew people with lots of money and lots of people withlittle or no money?

• Equality: Does it surprise you that the evidence fromÇatalhöyük suggests men and women lived very simi-lar lives and things were more or less equal betweenthem?

• Privacy: Çatalhöyük’s homes had no windows, justone main room, and an entrance from the roof! It’sperfect, don’t you think?

In the example below, we apply the BoC pattern andpresent the conversation episode on the first theme, which islinked to the interpretation of the evidence of burials found atÇatalhöyük. Figure 4 depicts an excerpt of an actual conversa-tion, using a variation of the bot’s questions and statements.The tone of the bot is kept informal, incorporating also a bit

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 7

Figure 3: The branding of the ChatÇat bot.

of witty ChatÇat personality, images and emojis.

• Intro - Provocation (A):Bot: Surely, you have people buried under your floors?User: [yes, no, never, I have, I haven’t, I would, Iwouldn’t, not, no way, sure, surely not, huh?, etc.]

• If user responds positively:Bot (Yes-1): I thought so! Do you have lots of peopleburied in your house?User: [Any response, e.g., No, Yes, Just the one, OMGthousands...]Bot (Yes-2): Do you plan on being buried in the house?User: [Any response]Bot (Testing conviction): I’d like to be kept in a house.I think it shows that people cared about you. Don’t youthink so?[continue to Finale]

• If user responds negatively:Bot (No-1):Well, where do you bury them then?User: [cemetery, graveyard, cremated, cremated athome... If users mention any element of home, house,etc. the thread continues with the final positive re-sponse, in this case Yes-2]Bot (No-2): Why would you put them so far away?Don’t you want them close to you? Where you can beconnected?User: [Any response][continue to Finale]

• If user responds ambiguously:Bot: Seriously, don’t you bury people in your houses?User: [Positive (see positive stream), Negative (seenegative stream), Ambiguous]

• If user remains ambiguous:Bot: I don’t get what’s so confusing. We buried peoplein our houses to show we cared. [continue to Finale]

• Finale (Statement of conviction):Bot: It is easy to forget when burial places seem so faraway but people live and work above the dead everyday. At Çatalhöyük we buried our loved ones in placeswhere they could remain a part of our daily lives. It isthrough our close relationship with the dead that westayed connected to our past.

6 IMPLEMENTATIONSince our goal was not to advance bot technology but toexplore and design a form of interaction that provokes usersto step out of their comfort zone, we deliberately decidedagainst developing our BoCwith sophisticated AI technology.Instead, we chose to implement a rule-based chatbot. Ourprimary reason was the need to have control over the UX,to construct a guided conversational approach. A rule-basedsystem was deemed sufficient to test if such an approach canactually evoke an emotional reaction.We chose Facebook’s Messenger platform for two main

reasons. Firstly, it is the largest and fastest growing mes-saging platform, with a wide user base [45]; secondly, it iseasy to author due to the broader set of tools available todevelopers.Each theme, as implemented, requires an interaction of

approximately 3 to 4 minutes with the bot. Nevertheless,the pattern can be extended to include multiple figure-8interactions in sequence in a conversational episode, if longerengagements are desired.

7 REVISITING THE BOT OF CONVICTIONSix months after the first formative studies that were carriedout to refine the design of the interaction with the earlier,primarily informational bot, we contacted almost all of the27 participants to see whether they were willing and able tospend some time testing the BoC. Nine (2 men and 7 women)of them responded positively and were either sent the linkto ChatÇat or were observed using it in person.Participants were encouraged to chat freely with the bot

in order to refresh their memory, but only for a few minutes,since they were presumably already familiar with it. Theywere then instructed to click on the button or type “Intrigueme”, for the BoC to kick in. After their experience, they wereasked to respond to a questionnaire. We also asked users ifthey would be willing to be interviewed about their experi-ence so that we could follow up directly with self-selectedindividuals whose logs we studied and whose interactionssurprised or otherwise interested us. We extended the ques-tionnaire used in our previous formative evaluation with fivenew questions focusing on the nature of the dialogue andlarger conceptual issues regarding perspective-taking andchallenging of users’ assumptions.

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 8

Figure 4: Excerpt from a conversation with ChatÇat.

To date, we have managed to obtain a full set of data (logs,questionnaire and interview responses) from 5 participants.In terms of usability, no major conversational breakdownoccurred, confirming that the pattern is sound and allowingus to focus on responses related to perspective-taking.The initial, direct “provocation” of the bot seemed to ef-

fectively engage the attention of the users and subsequentlyreinforce the point with follow-up questions. “I got a mini-shock, surprised I’d say, with this question coming out of the

blue ‘How would you feel if your grandmother was buried un-der your bed?’” (Steve). Users were pleasantly surprised andseemed to genuinely reflect on how the “radical” opinions ofthe bot actually revealed current preconceptions about thepast while also demonstrating that the same basic humanneeds and emotions continue to drive our own beliefs andpractices today.The bot as a conversation partner seems to take the ini-

tiative and challenge the user, trying to promote its point ofview and to make the user reflect, through a series of ques-tions: “I liked that the bot was asking me questions that werea bit provocative and caught my interest” (Vicky). Havingthe bot asking the questions in this way appears to workvery much in favor of promoting the illusion that the user istalking to an actual intelligent agent, a somewhat stronglyopinionated one perhaps, but still intelligent: “I really likedit, I didn’t expect to like it because I’m usually too cynical,like ‘Oh it’s just a machine’. But I think we became friendswith Chatcat” (Irene). The user is asked to respond and theresponses are seemingly taken into account, but the chat-bot, as a true conversational partner at times will seem tocare more about expressing its own opinion than listeningto the opinion of the user. Thus, the bot transforms froma mere neutral information provider to a rather stubbornconversation partner. This subtle transfer of control of thedialogue from the user to the chatbot seemingly works tofoster respect on the part of the user towards the bot, thuspromoting a deeper mental and emotional engagement inthe dialogue.

8 DISCUSSIONFrom our perspective, the approach of many chatbots used to-day is not necessarily productive for facilitating conversation,let alone genuine and extended dialogue [10]. In the contextin which we are working (cultural heritage), without mech-anisms to foster reflective debate and action, bots are littlemore than simplistic customer service lines or relentless in-formation providers with the potential to worsen cultural di-vides and reinforce problematic-but prevalent-contemporarypractices of nationalistic appropriation, racial bias [37], re-actionary populism, and imperialism. Substantial audienceresearch in the cultural sector clearly demonstrates that her-itage sites are places where people often come purposefullyto change their minds and are open to transformation [25],suggesting the potential for chatbots to provoke such action.However, beyond heritage, the same concerns for fosteringrespectful dialogue and argumentation leading to construc-tive social change in the world today are of increasing ur-gency [41].Our intention is not to design an affective bot that rec-

ognizes users’ emotional states and displays or respondsdirectly to these. Rather we aim to define simple means by

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 9

which a bot can provoke processes of critical reflection andaction. In this sense, the work described here focuses ondesigning, testing and refining conversational patterns forinterested practitioners to create socially beneficial changethrough dialogue around topics of broad public concern;and thus contributing to making critical design in HCI moreapproachable. Our work subscribes to the perspective chang-ing, dialogical framing of Bardzells’ (re)definition of “criticaldesign” [2] and is informed by Bardzell et al’s reflectionson designing for provocativeness [3]. In addition, our ap-proach to ‘critical reflection’ loosely relates to the modelfor historical empathy [8] where users are guided throughactivities designed to facilitate historical contextualization,perspective-taking and affective connection.

One of the positive aspects of our approach is that it guidesthe conversation in such a way that it is not easy to de-rail.Detecting the variations of a user’s yes and no responsesis relatively straightforward, while classifying all other re-sponses as ambiguous makes the pattern error-proof to agreat extent. The simplicity of the implementation thus be-comes one of its main strengths, especially for the fields ofheritage and education where many institutions do not havethe funding to acquire expensive solutions. Understandinghow the pattern works naturally leads to the design of a BoC,which requires nothing more than access to a text editor toimplement.

A conversation ‘episode’ comprises a user’s participationin a chat with the BoC. Our intended actions are multi-tiered,starting with the most basic: the user responds to the bot, andideally responds to the whole sequence of the chat, leadingto its conclusion, and ultimately to a new chat or continuedinteraction with the bot’s online offerings. Such actions, ifperformed by the user, would suggest the efficacy of the bot‘sconditions in provoking a reply.

The next tier entails the user demonstrating evidence of areconsideration of their original point of view through theirchat responses. Such actions, if evidenced in the inputtedtext and through associated evaluation (e.g., interviews, ques-tionnaires) would suggest the efficacy of the bot’s conditionsin provoking reflection or alternative perspective-taking.

From here, the next tier entails the user taking some formof action beyond the episode itself, suggesting the efficacyof the bot’s conditions in provoking transformation. In otherwords, the user’s interactions with the bot lead to changein their future ways of thinking, conversing with others oracting on the world.

Evidently, each tier requires a different assessment frame,as the most basic tier can be counted or quantified: ‘yes--user responded–chat continued’; ‘no-user left chat’. Thesecond tier requires discursive analysis of chat text and asso-ciated qualitative data collection (as we have done throughinterviews and surveys), wherein patterns of responses can

potentially be deduced with a focus on looking for changein the user’s point of view. The third tier requires a morelongitudinal evaluation approach, e.g. follow-up via user re-port (survey, interview), which again may be analyzed forpatterns in behavior, and which must appreciate that tyinghuman behaviors directly back to the influences of the botwill be challenging and necessarily open to interpretation,as with all affective practice.In its first incarnation, our BoC appears to confirm the

promise for rule-based chat patterns to provoke perspective-taking and the challenging of user assumptions. Understand-ing of its potential for transformational change now dependson wider development and evaluation.

Limitations and Further WorkA number of limitations were encountered in the courseof this work, pointing to several possible future directionsthat could be pursued on an empirical, methodological, andpractical level. Most importantly, in terms of evaluation, thestudy relating to the BoC was limited with respect to thenumber of participants and requires further attention to itseffectiveness in natural settings, longitudinally, and in termsof the questions posed at the outset. Furthermore, the evalu-ation instruments of post-experience self-reporting, throughquestionnaires and semi-structured interviews, present limi-tations [18]. Other researchers [47] and for other technolo-gies [27] [42] have identified the limitations presented bytraditional methods when attempting to capture users’ inter-action in unfolding, in-the-moment activities. We are alreadyexploring ways of embedding evaluation in the experience it-self, e.g. by weaving it intoMessenger in a relatively seamlessmanner. Nevertheless, as the issues pertaining to evaluationmethods range beyond the scope of this paper, we have keptthem out of the discussion and plan to address them in sub-sequent studies. Beyond evaluation, the BoC has potential asa dialogue facilitator in multi-user contexts. We are alreadyexploring such potential in related work in informal educa-tion contexts, with preliminary results showing the fosteringof historical empathy among middle school-aged users [30].

ContributionsUX and interaction designers increasingly have to wrestlewith the complex problem of designing digital encountersthat have relevance for users. In this space, we believe thatour contributions are threefold:

• Adding to the corpus of theoretical design considera-tions [37] concerning interactions with conversationalagents. If critical design and reflection should be a coretechnology design outcome of HCI [34] [2] [39], a keygoal of this work is to contribute towards creating theconditions in a user-to-bot interaction episode which

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 10

have the intention of soliciting specific intended ac-tions from participants.

• Proposing a design methodology that adopts a usercentered participatory approach, to make chatbots andconversational agents more relevant to their users.

• Ultimately contributing to the design of digital sys-tems that evoke meaningful interaction, envisioning aworld where HCI can become the impetus for personaltransformation and social change.

We have presented the design of a human-bot conver-sational experience that relies on a relatively simple set ofstrategies to facilitate meaning making. Although instanti-ated in a very specific cultural heritage setting, we believethat our framework is replicable in other contexts and anurgency for fostering critical dialogue in the world today.

ACKNOWLEDGMENTSThis work is part of the EMOTIVE project, which has re-ceived funding from the European Union’s Horizon 2020research and innovation programme under grant agreementNo. 727188. The authors would like to thank the EMOTIVEteam and Dr Vasilis Vlachokyriakos for their comments andinsightful conversations on drafts of this paper. We also wishto thank the users that participated in our formative studies,and our anonymous reviewers.

REFERENCES[1] Auckland Art Gallery. 2018. Auckland Art Gallery’s new chatbot: art-

ificial intelligence. http://www.scoop.co.nz/stories/CU1805/S00203/auckland-art-gallerys-new-chatbot-art-ificial-intelligence.htm Lastaccessed 31 December 2018.

[2] Jeffrey Bardzell and Shaowen Bardzell. 2013. What is "critical" aboutcritical design?. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems - CHI ’13. ACM Press, New York, NewYork, USA, 3297. https://doi.org/10.1145/2470654.2466451

[3] Shaowen Bardzell, Jeffrey Bardzell, Jodi Forlizzi, John Zimmerman, andJohn Antanitis. 2012. Critical design and critical theory: the challengeof designing for provocation. In Proceedings of the Designing InteractiveSystems Conference on - DIS ’12. ACM Press, New York, New York, USA,288. https://doi.org/10.1145/2317956.2318001

[4] Justine Cassell. 2001. Embodied conversational agents: representationand intelligence in user interfaces. AI Magazine 22, 4 (2001), 67–83.https://doi.org/10.1609/aimag.v22i4.1593

[5] Justine Cassell, Tim Bickmore, Lee Campbell, Hannes Vihjalmsson, andHao Yan. 2000. Human conversation as a system framework: designingembodied conversational agents. In Embodied conversational agents.MIT Press, Cambridge, MA, USA, Chapter 2, 29–62.

[6] Dot - Akron Art Museum. 2018. Dot - Akron Art Mu-seum guide. https://akronartmuseum.org/calendar/connect-with-dot-launch-party/12829 Last accessed 31 December2018.

[7] Anthony Dunne and Fiona Raby. 2001. Design Noir: The Secret Life ofElectronic Objects. Birkhäuser, Basel, Switzerland.

[8] Jason Endacott and Sarah Brooks. 2013. An Updated Theoretical andPractical Model for Promoting Historical Empathy. Social Studies

Research and Practice 8, 1 (2013), 41–58. http://www.socstrpr.org/wp-content/uploads/2013/04/MS_06482_no3.pdf

[9] Asbjørn Følstad and Petter Bae Brandtzæg. 2017. Chatbots and thenew world of HCI. interactions 24, 4 (jun 2017), 38–42. https://doi.org/10.1145/3085558

[10] Smithsonian Institution & Museweb Foundation. 2016. StorytellingToolkit - Facilitated Dialogue. Technical Report. 21 pages. https://museumonmainstreet.org/sites/default/files/facilitated_dialogue.pdf

[11] Katrina Gargett. 2018. Re-thinking the guided tour: co-creation, dialogueand practices of facilitation at York Minster. MA Thesis. University ofYork.

[12] Avelino J. Gonzalez, James R. Hollister, Ronald F. DeMara, Jason Leigh,Brandan Lanman, Sang-Yoon Lee, Shane Parker, Christopher Walls,Jeanne Parker, Josiah Wong, Clayton Barham, and Bryan Wilder. 2017.AI in Informal Science Education: Bringing Turing Back to Life toPerform the Turing Test. International Journal of Artificial Intelli-gence in Education 27, 2 (jun 2017), 353–384. https://doi.org/10.1007/s40593-017-0144-1

[13] Shawn Graham. 2017. An Introduction to Twitter Bots with Tracery.https://programminghistorian.org/en/lessons/intro-to-twitterbots

[14] Tom Hennes. 2002. Rethinking the Visitor Experience: TransformingObstacle into Purpose. Curator: The Museum Journal 45, 2 (apr 2002),105–117. https://doi.org/10.1111/j.2151-6952.2002.tb01185.x

[15] Anne Frank House. 2017. Anne Frank House bot for Messengerlaunch. https://www.annefrank.org/en/about-us/news-and-press/news/2017/3/21/anne-frank-house-launches-bot-messenger/ Last ac-cessed 31 December 2018.

[16] Akrivi Katifori, Maria Roussou, Sara Perry, George Drettakis, SebastianVizcay, and Julien Philip. 2018. The EMOTIVE Project - Emotive virtualcultural experiences through personalized storytelling. In EuroMed2018, International Conference on Cultural Heritage. Lemessos, Cyprus.

[17] Mark Katrikh. 2018. Creating Safe(r) Spaces for Visitors and Staff inMuseum Programs. Journal of Museum Education 43, 1 (jan 2018), 7–15.https://doi.org/10.1080/10598650.2017.1410673

[18] A. Baki Kocaballi, Liliana Laranjo, and Enrico Coiera. 2018. Mea-suring User Experience in Conversational Interfaces: A Compari-son of Six Questionnaires. In Proceedings of the 32Nd InternationalBCS Human Computer Interaction Conference (HCI ’18). BCS Learn-ing & Development Ltd., Swindon, UK, Article 21, 12 pages. https://doi.org/10.14236/ewic/HCI2018.21

[19] Stefan Kopp, Christian Becker, and IpkeWachsmuth. 2006. The VirtualHuman Max - Modeling Embodied Conversation. In KI 2006 - DemoPresentation, Extended Abstracts. 19–22.

[20] Stefan Kopp, Lars Gesellensetter, Nicole C. Krämer, and IpkeWachsmuth. 2005. A Conversational Agent as Museum Guide âĂŞDesign and Evaluation of a Real-World Application. In IntelligentVirtual Agents. IVA 2005. Lecture Notes in Computer Science, vol 3661,T. Panayiotopoulos, J. Gratch, Ruth Aylett, Ballin Dan, Olivier Patrick,and T. Rist (Eds.). Springer Berlin Heidelberg, 329–343. https://doi.org/10.1007/11550617_28

[21] Peter M. Krafft, Michael Macy, and Alex "Sandy" Pentland. 2017. Botsas Virtual Confederates. In Proceedings of the 2017 ACM Conference onComputer Supported Cooperative Work and Social Computing - CSCW’17. ACM Press, New York, New York, USA, 183–190. https://doi.org/10.1145/2998181.2998354

[22] H. Chad Lane, Clara Cahill, Susan Foutz, Daniel Auerbach, Dan Noren,Catherine Lussenhop, and William Swartout. 2013. The Effects of aPedagogical Agent for Informal Science Education on Learner Behav-iors and Self-efficacy. In Artificial Intelligence in Education. AIED 2013.Lecture Notes in Computer Science, vol 7926, H. Chad Lane, Kalina Yacef,JackMostow, and P. Pavlik (Eds.). Springer BerlinHeidelberg,Memphis,TN, USA, 309–318. https://doi.org/10.1007/978-3-642-39112-5_32

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 11

[23] Q. Vera Liao, Werner Geyer, Muhammed Mas-ud Hussain, PraveenChandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso,Dakuo Wang, Michael Muller, and N. Sadat Shami. 2018. All Workand no Play? Conversations with a Question-and-Answer Chatbot inthe Wild. In Proceedings of the 2018 CHI Conference on Human Factorsin Computing Systems - CHI ’18. ACM, New York, NY, USA, 1–13.https://doi.org/10.1145/3173574.3173577

[24] Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA":The Gulf between User Expectation and Experience of ConversationalAgents. In Proceedings of the 2016 CHI Conference on Human Factors inComputing Systems - CHI ’16. ACM Press, New York, New York, USA,5286–5297. https://doi.org/10.1145/2858036.2858288

[25] Bernadette Lynch. 2013. Reflective debate, radical transparency andtrust in museums. Museum Management and Curatorship 28, 1 (2013),1–13.

[26] Matt Malpass. 2013. Between Wit and Reason: Defining As-sociative, Speculative, and Critical Design in Practice. Designand Culture 5, 3 (nov 2013), 333–356. https://doi.org/10.2752/175470813X13705953612200

[27] Timothy Marsh, Peter Wright, and Shamus P. Smith. 2001. Evalua-tion for the design of experience in virtual environments: modelingbreakdown of interaction and illusion. Cyberpsychology & behavior: theimpact of the Internet, multimedia and virtual reality on behavior and so-ciety 4, 2 (2001), 225–238. https://doi.org/10.1089/109493101300117910

[28] Nikita Mattar and Ipke Wachsmuth. 2014. Let’s Get Personal. InHuman-Computer Interaction. Advanced Interaction Modalities andTechniques. HCI 2014. Lecture Notes in Computer Science, vol 8511.Springer, Cham, 450–461. https://doi.org/10.1007/978-3-319-07230-2_43

[29] John McCarthy, Peter Wright, Jayne Wallace, and Andy Dearden.2006. The experience of enchantment in human-computer inter-action. Personal and Ubiquitous Computing 10, 6 (2006), 369–378.https://doi.org/10.1007/s00779-005-0055-2

[30] Sierra McKinney. 2018. Generating pre-historical empathy in classrooms.Master’s thesis. University of York.

[31] Michael Minge and Manfred Thüring. 2018. Hedonic and pragmatichalo effects at early stages of User Experience. International Journalof Human-Computer Studies 109 (jan 2018), 13–25. https://doi.org/10.1016/j.ijhcs.2017.07.007

[32] Elahe Paikari and André van der Hoek. 2018. A framework for un-derstanding chatbots and their future. In Proceedings of the 11th In-ternational Workshop on Cooperative and Human Aspects of SoftwareEngineering - CHASE ’18. ACM Press, New York, New York, USA, 13–16.https://doi.org/10.1145/3195836.3195859

[33] Sara Perry. 2018. The Enchantment of the Archaeological Record.In 24th Annual Meeting of the European Association of Archaeologists.European Association of Archaeologists, Barcelona, Spain.

[34] James Pierce, Phoebe Sengers, Tad Hirsch, Tom Jenkins, William Gaver,and Carl DiSalvo. 2015. Expanding and Refining Design and Criticalityin HCI. In Proceedings of the 33rd Annual ACM Conference on HumanFactors in Computing Systems - CHI ’15. ACM Press, New York, NewYork, USA, 2083–2092. https://doi.org/10.1145/2702123.2702438

[35] Maria Roussou and Akrivi Katifori. 2018. Flow, Staging, Wayfinding,Personalization: Evaluating User Experience with Mobile MuseumNarratives. Multimodal Technologies and Interaction 2, 2 (jun 2018), 32.https://doi.org/10.3390/MTI2020032

[36] Mark Sample. 2014. A protest bot is a bot so specific you can’t mistakeit for bullshit: A Call for Bots of Conviction. http://bit.ly/2F3fYGO

[37] Ari Schlesinger, Kenton P. O’Hara, and Alex S. Taylor. 2018. Let’sTalk About Race: Identity, Chatbots, and AI. In Proceedings of the 2018CHI Conference on Human Factors in Computing Systems - CHI ’18.ACM Press, New York, New York, USA, 1–14. https://doi.org/10.1145/

3173574.3173889[38] M. Schroder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M.

ter Maat, G. McKeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller,E. de Sevin, M. Valstar, and M. Wollmer. 2012. Building AutonomousSensitive Artificial Listeners. IEEE Transactions on Affective Computing3, 2 (apr 2012), 165–183. https://doi.org/10.1109/T-AFFC.2011.34

[39] Phoebe Sengers, Kirsten Boehner, Shay David, and Joseph ’Jofish’ Kaye.2005. Reflective design. In Proceedings of the 4th decennial conferenceon Critical computing between sense and sensibility - CC ’05. ACMPress, New York, New York, USA, 49. https://doi.org/10.1145/1094562.1094569

[40] Samira Shaikh. 2017. A persuasive virtual chat agent based on so-ciolinguistic theories of influence. AI Matters 3, 2 (jul 2017), 26–27.https://doi.org/10.1145/3098888.3098899

[41] Walter Sinnott-Armstrong. 2018. Think Again: How to Reason andArgue. Penguin, London, UK.

[42] Mel Slater. 2004. How Colorful Was Your Day? Why Question-naires Cannot Assess Presence in Virtual Environments. Presence:Teleoperators and Virtual Environments 13, 4 (aug 2004), 484–493.https://doi.org/10.1162/1054746041944849

[43] Laurajane Smith. 2016. Changing views? Emotional intelligence, reg-isters of engagement, and the museum visit. In Museums as Sites ofHistorical Consciousness: Perspectives on museum theory and practicein Canada, Vivienne Gosselin and Phaedra Livingstone (Eds.). UBCPress, Vancouver, Canada, Chapter 6, 101–121.

[44] Barbara J. Soren. 2009. Museum experiences that change visitors.MuseumManagement and Curatorship 24, 3 (sep 2009), 233–251. https://doi.org/10.1080/09647770903073060

[45] Statista. 2018. Number of monthly active Facebook Messenger usersfrom April 2014 to September 2017 (in millions). https://www.statista.com/statistics/417295/facebook-messenger-monthly-active-users/Last accessed 31 December 2018.

[46] William Swartout, David Traum, Ron Artstein, Dan Noren, Paul De-bevec, Kerry Bronnenkant, Josh Williams, Anton Leuski, ShrikanthNarayanan, Diane Piepol, Chad Lane, Jacquelyn Morie, Priti Aggarwal,Matt Liewer, Jen-Yuan Chiang, Jillian Gerten, Selina Chu, and KyleWhite. 2010. Ada and Grace: Toward Realistic and Engaging VirtualMuseum Guides. In IVA 2010, J. Allbeck (Ed.). Springer-Verlag BerlinHeidelberg, 286–300. http://ict.usc.edu/pubs/adaandgrace.pdf

[47] Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed.2018. The Ethnobot: Gathering Ethnographies in the Age of IoT. InProceedings of the 2018 CHI Conference on Human Factors in ComputingSystems - CHI ’18. ACM Press, New York, New York, USA, 1–13. https://doi.org/10.1145/3173574.3174178

[48] The House Museums of Milan. 2016. Di Casa in casa adventour. https://www.facebook.com/dicasaincasagame/ Last accessed 31 December2018.

[49] Angeliki Tzouganatou. 2017. Chatbot Experience for ÇATALHÖYÜK.Master’s thesis. University of York.

[50] Stavros Vassos, Eirini Malliaraki, Federica dal Falco, Jessica Di Mag-gio, Manlio Massimetti, Maria Giulia Nocentini, and Angela Testa.2016. Art-Bots: Toward Chat-Based Conversational Experiences inMuseums. In Interactive Storytelling. 9th International Conferenceon Interactive Digital Storytelling, ICIDS 2016, Frank Nack and An-drew S. Gordon (Eds.). Los Angeles, CA, USA, 433–437. https://doi.org/10.1007/978-3-319-48279-8_43

[51] Astrid M. von der Pütten, Nicole C. Krämer, Jonathan Gratch, andSin-Hwa Kang. 2010. “It doesn’t matter what you are!” Explainingsocial effects of agents and avatars. Computers in Human Behavior 26,6 (nov 2010), 1641–1650. https://doi.org/10.1016/j.chb.2010.06.012

[52] Margaret Wetherell. 2012. Affect and Emotion (1st ed.). Sage Publica-tions Ltd, London, UK. 192 pages.

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 12

[53] Margaret Wetherell, Laurajane Smith, and Gary Campbell. 2018. Intro-duction: Affective heritage practices. In Emotion, Affective Practices,and the Past in the Present, Laurajane Smith, Margaret Wetherell, andGary Campbell (Eds.). Routledge, London, 1–21.

[54] Peter Wright and John McCarthy. 2008. Empathy and experience inHCI. In Proceeding of the twenty-sixth annual CHI conference on Humanfactors in computing systems - CHI ’08. ACM Press, New York, NewYork, USA, 637. https://doi.org/10.1145/1357054.1357156

CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

Paper 627 Page 13