balancing objectives in counseling conversations...
TRANSCRIPT
Balancing Objectives in Counseling Conversations:Advancing Forwards or Looking Backwards
Justine ZhangCornell University
Cristian Danescu-Niculescu-MizilCornell University
Abstract
Throughout a conversation, participants makechoices that can orient the flow of the inter-action. Such choices are particularly salientin the consequential domain of crisis counsel-ing, where a difficulty for counselors is balanc-ing between two key objectives: advancing theconversation towards a resolution, and empa-thetically addressing the crisis situation.
In this work, we develop an unsupervisedmethodology to quantify how counselors man-age this balance. Our main intuition is that ifan utterance can only receive a narrow rangeof appropriate replies, then its likely aim isto advance the conversation forwards, towardsa target within that range. Likewise, an ut-terance that can only appropriately follow anarrow range of possible utterances is likelyaimed backwards at addressing a specific sit-uation within that range. By applying this in-tuition, we can map each utterance to a contin-uous orientation axis that captures the degreeto which it is intended to direct the flow of theconversation forwards or backwards.
This unsupervised method allows us to char-acterize counselor behaviors in a large datasetof crisis counseling conversations, where weshow that known counseling strategies intu-itively align with this axis. We also illustratehow our measure can be indicative of a conver-sation’s progress, as well as its effectiveness.
1 Introduction
Participants in a conversation constantly shapethe flow of the interaction through their choices.In psychological crisis counseling conversations,where counselors support individuals in mentaldistress, these choices arise in uniquely complexand high-stakes circumstances, and are reflectedin rich conversational dynamics (Sacks, 1992). Assuch, counseling is a valuable context for computa-tionally modeling conversational behavior (Atkins
when i tell my mom about the bullies she just ignores me
Have you confided to anyone else about this?
yeah there’s my sister… she just tells me to get over it
That sounds so frustrating, you deserve to be listened to.
t0
t1 t2
c1 c2
Figure 1: Two possible exchanges in a counseling con-versation, illustrating key objectives that a counselormust balance: c1 aims to advance the conversation to-wards a discussion of possible confidants; c2 aims to ad-dress the emotion underlying the preceding utterance.
et al., 2014; Althoff et al., 2016; Perez-Rosas et al.,2018; Zhang et al., 2019). Modeling the conversa-tional choices of counselors in this endeavor is animportant step towards better supporting them.
Counselors are driven by several objectives thatserve the broader goal of helping the individualin distress; two key objectives are exemplified inFigure 1.1 The counselor must advance a conver-sation towards a calmer state where the individ-ual is better equipped to cope with their situation(Mishara et al., 2007; Sandoval et al., 2009): in c1,the counselor prompts the individual to brainstormoptions for social support. The counselor mustalso empathetically address what was already said,“coming to an empathic understanding” of the indi-vidual (Rogers, 1957; Hill and Nakayama, 2000):in c2, the counselor validates feelings that the indi-vidual has just shared.
Balancing both objectives is often challenging,and overshooting in one direction can be detrimen-tal to the conversation. A counselor who leans toomuch on advancing forwards could rush the con-versation at the expense of establishing an empa-thetic connection; a counselor who leans too muchbackwards, on addressing what was already said,may fail to make any progress.
1These examples are derived from material used to traincounselors in our particular setting, detailed in Section 2.
In this work, we develop a method to examinecounselor behaviors as they relate to this balancingchallenge. We quantify the relative extent to whichan utterance is aimed at advancing the conversa-tion, versus addressing existing content. We thusmap each utterance onto a continuous backwards-forwards axis which models the balance of theseobjectives, and refer to an utterance’s position onthis axis as its orientation.
At an intuitive level, our approach considers therange of content that is expected to follow or pre-cede a particular utterance. For an utterance like c1that aims to advance the conversation towards anintended target, we would expect a narrow rangeof appropriate replies, concentrated around thattarget (e.g., suggestions of possible confidants).We would likewise expect an utterance like c2 thataims to address a previously-discussed situation toonly be an appropriate reply for a narrow range ofpossible utterances, concentrated around that spe-cific type of situation (e.g., disclosures of negativefeelings). Starting from this intuition, we developan unsupervised method to quantify and comparethese expected forwards and backwards ranges forany utterance, yielding our orientation measure.
Using this measure, we characterize counselorbehaviors in a large collection of text-message con-versations from a crisis counseling service, whichwe accessed in collaboration with the service andwith the participants’ consent. We show how ori-entation meaningfully distinguishes between keyconversational strategies that counselors are taughtduring their training. We also show that our mea-sure tracks a conversation’s progress and can sig-nal its effectiveness, highlighting the importanceof balancing the advancing and addressing objec-tives, and laying the basis for future inquiries inestablishing potential causal effects.
In summary, we develop an unsupervisedmethodology that captures how counselors bal-ance the conversational objectives of advancingand addressing (Section 4), apply and validateit in a large dataset of counseling conversations(Section 5), and use it to investigate the rela-tion between a counselor’s conversational behav-ior and their effectiveness (Section 5.4). Whileour method is motivated by a salient challenge incounseling, we expect similar balancing problemsto recur in other conversational settings where par-ticipants must carefully direct the flow of the inter-action, such as court trials and debates (Section 6).
2 Setting: Counseling Conversations
We develop our method in the context of Cri-sis Text Line, a crisis counseling platform whichprovides a free 24/7 service for anyone in men-tal crisis—henceforth texters—to have one-on-one conversations via text message with affiliatedcounselors. We accessed a version of this collec-tion, with over 1.5 million conversations, in col-laboration with the platform and with the consentof the participants. The data was scrubbed of per-sonally identifiable information by the platform.2
These conversations are quite substantive, averag-ing 25 messages with 29 and 24 words per coun-selor and texter message, respectively.
In each conversation, a crisis counselor’s high-level goal is to guide the texter towards a calmermental state. In service of this goal, all counselorsfirst complete 30 hours of training provided by theplatform, which draws on past literature in coun-seling to recommend best practices and conversa-tional strategies. The first author also completedthe training to gain familiarity with the domain.
While the platform offers guidance to coun-selors, their task is inevitably open-ended, giventhe emotional complexity of crisis situations. Assuch, the counselors are motivated by an explicitgoal that structures the interaction, but they face achallenging flexibility in choosing how to act.
3 Background and Related Work
We now describe the conversational challenge ofbalancing between advancing the conversation for-wards or addressing what was previously said. Ourdescription of the challenge and our computationalapproach to studying it are informed by literaturein counseling, on the platform’s training materialand on informal interviews with its staff.A conversational balance. A crisis counselormust fulfill multiple objectives in their broadergoal of helping a texter. One objective is guidingthe texter through their initial distress to a calmermental state (Mishara et al., 2007; Sandoval et al.,2009), as in Figure 1, c1. Various strategies thataim to facilitate this advancing process are taughtto counselors during training: for instance, a coun-selor may prompt a texter to identify a goal or cop-
2The data can be accessed by applying at https://www.crisistextline.org/data-philosophy/data-fellows/. The extensive ethical and privacyconsiderations, and policies accordingly implemented by theplatform, are detailed in Pisani et al. (2019).
ing mechanism (Rollnick and Miller, 1995). Assuch, they attempt to move the conversation for-wards, towards its eventual resolution.
The counselor must also engage with the tex-ter’s concerns (Rogers, 1957; Hill and Nakayama,2000), as in c2, via strategies that empatheticallyaddress what the texter has already shared (Roll-nick and Miller, 1995; Weger et al., 2010; Bodieet al., 2015). For instance, counselors are taughtto reflect, i.e., reframe a texter’s previous messageto convey understanding, or draw on what was saidto affirm the texter’s positive qualities. In doing so,the counselor looks backwards in the conversation.
Past work has posited the benefits of mixingbetween strategies that aim at either objective(Mishara et al., 2007). However, as the trainingacknowledges, striking this balance is challenging.Overzealously seeking to advance could cut shortthe process of establishing an empathetic connec-tion. Conversely, focusing on the conversation’spast may not help with eventual problem solving(Bodie et al., 2015), and risks stalling it. A tex-ter may start to counterproductively rehash or ru-minate on their concerns (Nolen-Hoeksema et al.,2008; Jones et al., 2009); indeed, prior psycholog-ical work has highlighted the thin line betweenproductive reflection and rumination (Rose et al.,2007; Landphair and Preddy, 2012).Orientation. To examine this balancing dynamic,we model the choices that counselors make at eachturn in a conversation. Our approach is to derivea continuous axis spanned by advancing and ad-dressing. We refer to an utterance’s position onthis axis, representing the relative extent to whichit aims at either objective, as its orientation Ω. Weinterpret a forwards-oriented utterance with posi-tive Ω as aiming to advance the conversation, anda backwards-oriented utterance with negative Ω asaiming to address what was previously brought up.In the middle, the axis reflects the graded way inwhich a counselor can balance between aims—forinstance, using something the texter has previouslysaid to help motivate a problem-solving strategy.Related characterizations. While we develop ori-entation to model a dynamic in counseling, weview it as a complement to other characterizationsof conversational behaviors in varied settings.
Prior work has similarly considered how utter-ances relate to the preceding and subsequent dis-course (Webber, 2001). Frameworks like center-ing theory (Grosz et al., 1995) aim at identify-
ing referenced entities, while we aim to more ab-stractly model interlocutor choices. Past work hasalso examined how interlocutors mediate a conver-sation’s trajectory through taking or ceding con-trol (Walker and Whittaker, 1990) or shifting topic(Nguyen et al., 2014); Althoff et al. (2016) consid-ers the rate at which counselors in our setting ad-vance across stages of a conversation. While theseactions can be construed as forwards-oriented, wefocus more on the interplay between forwards- andbackwards-oriented actions. A counselor’s objec-tives may also cut across these concepts: for in-stance, the training stresses the need for empa-thetic reflecting across all stages and topics.
Orientation also complements prior work on di-alogue acts, which consider various roles that ut-terances play in discourse (Mann and Thompson,1988; Core and Allen, 1997; Ritter et al., 2010;Bracewell et al., 2012; Rosenthal and McKeown,2015; Prabhakaran et al., 2018; Wang et al., 2019).In counseling settings, such approaches have high-lighted strategies like reflection and question-asking (Houck, 2008; Gaume et al., 2010; Atkinset al., 2014; Can et al., 2015; Tanana et al., 2016;Perez-Rosas et al., 2017, 2018; Park et al., 2019;Lee et al., 2019; Cao et al., 2019). Instead of mod-eling a particular taxonomy of actions, we modelhow counselors balance among the underlying ob-jectives; we later relate orientation to these strate-gies (Section 5). Most of these approaches useannotations or predefined labeling schemes, whileour method is unsupervised.
4 Measuring Orientation
We now describe our method to measure orienta-tion, discussing our approach at a high level be-fore elaborating on the particular operationaliza-tion. The code implementing our approach is dis-tributed as part of the ConvoKit library (Changet al., 2020), at http://convokit.cornell.edu.
4.1 High-Level Sketch
Orientation compares the extent to which an utter-ance aims to advance the conversation forwardswith the extent to which it looks backwards. Thus,we must somehow quantify how the utterance re-lates to the subsequent and preceding interaction.Naive attempt: direct comparison. As a naturalstarting point, we may opt for a similarity-basedapproach: an utterance that aims to address its pre-ceding utterance, or predecessor, should be similar
sounds frustrating
confided to anyone
ignoresjudges
laughs
doesn’t
becausejust
problem
ignorenothing sister
friend
counselorexpected predecessors: expected replies:
Figure 2: Words representative of replies and predeces-sors for utterances with two example phrasings, as ob-served in training data. Top row: observed replies to ut-terances with w1 span a narrower range than observedpredecessors (relative sizes of red and blue circles); w1
thus has smaller forwards-range −−→σw1 than backwards-range←−−σw1 (i.e., it is forwards-oriented, Ωw1 > 0). Bot-tom row: observed predecessors to utterances with w2
span a narrower range than replies; w2 thus has smaller←−−σw2 than −−→σw2 (i.e., it is backwards-oriented Ωw2 < 0).
to it; an utterance that aims to advance the conver-sation should be similar to the reply that it prompts.In practice, having to make these direct compar-isons is limiting: an automated system could notcharacterize an utterance in an ongoing conversa-tion by comparing it to a reply it has yet to receive.
This approach also has important conceptualfaults. First, addressing preceding content in a con-versation is different from recapitulating it. For in-stance, counselors are instructed to reframe ratherthan outright restate a texter’s message, as in Fig-ure 1, c2. Likewise, counselors need not advancethe conversation by declaring something for thetexter to simply repeat; rather than giving spe-cific recommendations, counselors are instructedto prompt the texters to come up with copingstrategies on their own, as in c1. Further, textersare not bound to the relatively formal linguisticstyle counselors must maintain, resulting in clearlexical differences. Measuring orientation is hencea distinct task from measuring similarity.
Second, an utterance’s intent to advance neednot actually be realized. A counselor’s cues maybe rebuffed or misunderstood (Schegloff, 1987;Thomas, 1983): a texter could respond to c1by continuing to articulate their problem with t2.Likewise, a counselor may intend to address a tex-ter’s concerns but misinterpret them. To model thebalance in objectives that a counselor is aiming for,our characterization of an utterance cannot be con-tingent on its actual reply and predecessor.Our approach: characterizing expectations.We instead consider the range of replies we mightexpect an utterance to receive, or the range of pre-
decessors that it might follow. Intuitively, an ut-terance with a narrow range of appropriate repliesaims to direct the conversation towards a particu-lar target, moreso than an utterance whose appro-priate replies span a broader range.3 Likewise, anutterance that is an appropriate reply to only a nar-row range of possible predecessors aims to addressa particular situation. We draw on unlabeled dataof past conversations to form our expectations ofthese ranges, and build up our characterizations ofutterances from their constituent phrasings, e.g.,words or dependency-parse arcs.
The intuition for our approach is sketched inFigure 2. From our data, we observe that ut-terances containing confided to anyone generallyelicited replies about potential confidants (e.g., sis-ter, friend), while the replies that followed utter-ances with sounds frustrating span a broader, lesswell-defined range. As such, we have a strongerexpectation of what a reply prompted by a newutterance with confided to anyone might containthan a reply to a new utterance with sounds frus-trating. More generally, for each phrasing w, wequantify the strength of our expectations of itspotential replies by measuring the range spannedby the replies it has already received in the data,which we refer to as its forwards-range −→σw. Wewould say that confided to anyone has a smaller−→σwthan sounds frustrating, meaning that its observedreplies were more narrowly concentrated; this isrepresented as the relative size of the red regionson the right side of Figure 2.
In the other direction, we observe in our datathat sounds frustrating generally followed de-scriptions of frustrating situations (e.g., ignores,judges), while the range of predecessors to con-fided to anyone is broader. We thus have a strongerexpectation of the types of situations that new ut-terances with sounds frustrating would respond to,compared to new utterances with confided to any-one. For a phrasing w, we quantify the strengthof our expectations of its potential predecessorsby measuring its backwards-range ←−σw, spannedby the predecessors we’ve observed. As such,sounds frustrating has a smaller←−σw than confidedto anyone, corresponding to the relative size of theblue regions on the left side of Figure 2.
3Consider leading versus open-ended questions. Whenpeople ask leading questions, they intend to direct the inter-action towards specific answers they have in mind; when peo-ple ask open-ended questions, they are more open to whatanswers they receive and where the interaction is headed.
The relative strengths of our expectations in ei-ther direction then indicate the balance of objec-tives. If we have a stronger expectation of w’sreplies than of its predecessors—i.e., smaller −→σwthan ←−σw—we would infer that utterances with waim to advance the conversation towards a targetedreply more than they aim to address a particu-lar situation. Conversely, if we have stronger ex-pectations of w’s predecessors—i.e., smaller←−σw—we would infer that utterances with w aim to ad-dress the preceding interaction, rather than tryingto drive the conversation towards some target.
We thus measure orientation by comparing aphrasing’s forwards- and backwards-range. Theexpectation-based approach allows us to circum-vent the shortcomings of a direct comparison; wemay interpret it as modeling a counselor’s intentin advancing and addressing at each utterance(Moore and Paris, 1993; Zhang et al., 2017).
4.2 Operationalization
We now detail the steps of our method, which areoutlined in Figure 3. Formally, our input consistsof a set of utterances from counselors ci, and aset of utterances from texters ti, which we’veobserved in a dataset of conversations (Figure 3A).We note that each texter utterance can be a reply to,or a predecessor of, a counselor utterance (or both).We use this unlabeled “training data” to measurethe forwards-range −→σw, the backwards-range ←−σw(Figures 3B-D), and hence the orientation Ωw ofeach phrasing w used by counselors (Figure 3E).We then aggregate to an utterance-level measure.
For each counselor phrasing w, let−→Tw denote
the subset of texter utterances which are replies tocounselor utterances containing w (Figure 3A). Asdescribed above, the forwards-range−→σw quantifiesthe spread among elements of
−→Tw; we measure this
by deriving vector representations of these utter-ances
−→Uw (Figure 3B, detailed below), and then
comparing each vector in−→Uw to a central refer-
ence point −→uw (Figures 3C and 3D).4 Likewise,←−σw quantifies the similarity among elements of←−Tw, the set of predecessors to counselor utteranceswith w; we compute←−σw by comparing each corre-sponding vector in
←−Uw to a central point←−uw.
4Using a central reference point to calculate the forwards-range, as opposed to directly computing pairwise similaritiesamong replies in
−→Uw, allows us to account for the context
of w in the utterances that prompted these replies (via tf-idfweighting, as subsequently discussed).
A. Input: observed texter replies to counselor utterances
B. Derive vector representations of texter utterances
C. Derive central points D. Compute forwards-range
E. Compute orientation:
confided to anyone
yeah there’s my sister
i told my friend…
the school counselor…
ci: have you confided to anyone about this?
cj: I wonder if you’ve confided to anyone…
ck: have you confided to anyone recently?
…
…
… …
reply
reply
reply
reply
reply
SVD
whereis the cosine distancebetween and
where is thetf-idf weightof in
where is a tf-idf reweightedterm-document matrix of all texter utterances
low-dimensionalrepresentations of texter utterances in
example
Figure 3: Outline of steps to compute orientation Ωw
of phrasing w, as described in Section 4.2. Panels A-Dshow the procedure for computing forwards-range −→σw;the procedure for backwards-range←−σw is similar.
Deriving vector representations (Figure 3B). Toobtain vectors for each texter utterance, we con-struct X , a tf-idf reweighted term-document ma-trix where rows represent texter utterances andcolumns represent phrasings used by texters. Toensure that we go beyond lexical matches and cap-ture conceptual classes (e.g., possible confidants,frustrating situations), we use singular value de-composition to get X ≈ USV T . Each row of U isa vector representation ui of utterance ti in the in-duced low-dimensional space T.
−→Uw then consists
of the corresponding subset of rows of U (high-lighted in Figure 3B).Deriving central points (Figure 3C). For each w,we take the central point −→uw to be a weighted aver-age of vectors in
−→Uw. Intuitively, a texter utterance
ti with vector ui should have a larger contributionto −→uw if w is more prominent in the counselor ut-terance ci that preceded it. We let wi
w denote thenormalized tf-idf weight of w in ci, and use wi
w asthe weight of the corresponding vector ui. To prop-erly map the resultant weighted sum
∑wiwui into
T, we divide each dimension by the correspondingsingular value in S. As such, if ww is a vector ofweights wi
w, we can calculate the central point −→uw
of−→Uw as −→uw = wT
w
−→UwS
−1. In the other direction,we likewise compute←−uw = wT
w
←−UwS
−1.Forwards- and backwards-ranges (Figure 3D).We take the forwards-range −→σw of w to be the av-erage cosine distance from each vector in
−→Uw to
the center point −→uw. Likewise, we take ←−σw as theaverage distance from each vector in
←−Uw to←−uw.
Phrasing-level orientation (Figure 3E). Impor-tantly, since we’ve computed the forwards- andbackwards-ranges −→σw and ←−σw using distances inthe same space T, their values are comparable. Wethen compute the orientation of w as their differ-ence: Ωw =←−σw −−→σw.Utterance-level orientation. To compute the ori-entation of an utterance ci, we first compute theorientation of each sentence in ci as the tf-idfweighted average Ωw of its constitutent phrasings.Note that a multi-sentence utterance can orientin both directions—e.g., a counselor could con-catenate c2 and c1 from Figure 1 in a single ut-terance, addressing the texter’s previous utterancebefore moving ahead. To model this heterogene-ity, we consider both the minimum and maximumsentence-orientations in an utterance: Ωmin cap-tures the extent to which the utterance looks back-wards, while Ωmax captures the extent to which itaims to advance forwards.
5 Application to Counseling Data
We apply our method to characterize messagesfrom crisis counselors on the platform. We com-pute the orientations of the phrasings they use, rep-resented as dependency-parse arcs. We use a train-ing set of 351,935 texter and counselor messageseach, from a random sample of conversations omit-ted in subsequent analyses.5 Table 1 shows repre-sentative phrasings and sentences of different ori-entations.6 Around two-thirds of phrasings andsentences have Ω <0, echoing the importance ofaddressing the texter’s previous remarks.
In what follows, we analyze counselor behav-iors in terms of orientation, and illustrate howthe measure can be useful for examining conver-sations. We start by validating our method viatwo complementary approaches. In a subset ofsentences manually annotated with the counseling
5Further implementation details are listed in the appendix.6Example sentences are derived from real sentences in the
data, and modified for readability. The examples were chosento reflect common situations in the data, and were vetted bythe platform to ensure the privacy of counselors and texters.
strategies they exhibit, we show that orientationmeaningfully reflects these strategies (Section 5.1).At a larger scale, we show that the orientation ofutterances over the course of a conversation alignswith domain knowledge about counseling conver-sation structure (Section 5.2). We also find thatother measures for characterizing utterances arenot as rich as orientation in capturing counselingstrategies and conversation structure (Section 5.3).Finally, we show that a counselor’s orientation in aconversation is tied to indicators of their effective-ness in helping the texter (Section 5.4).
5.1 Validation: Counseling Strategies
Even though it is computed without the guidanceof any annotations, we expect orientation to mean-ingfully reflect strategies for advancing or address-ing that crisis counselors are taught. The firstauthor hand-labeled 400 randomly-selected sen-tences with a set of pre-defined strategies derivedfrom techniques highlighted in the training mate-rial. We note example sentences in Table 1 whichexemplify each strategy, and provide more exten-sive descriptions in the appendix.
Figure 4A depicts the distributions of ori-entations across each label, sorted from mostbackwards- to most forwards-oriented. We findthat the relative orientation of different strategiescorroborates their intent as described in the liter-ature. Statements reflecting or affirming whatthe texter has said to check understanding or con-vey empathy (characterized by phrasings like to-tally normal) tend to be backwards-oriented; state-ments prompting the texter to advance towardsproblem-solving (e.g., [what] has helped) aremore forwards-oriented. Exploratory queries formore information on what the texter has alreadysaid (e.g., happened to make) tend to have mid-dling orientation (around 0). The standard devia-tion of orientations over messages within most ofthe labels is significantly lower than across labels(bootstrapped p< .05, solid circles), showing thatorientation yields interpretable groupings of mes-sages in terms of important counseling strategies.
The measure also offers complementary infor-mation. For instance, we find sentences that aren’taccounted for by pre-defined labels, but still mapto interpretable orientations, such as backwards-oriented examples assuaging texter concerns aboutthe platform being a safe space to self-disclose.
Orientation Example phrasings Example sentences
Backwards-oriented(bottom 25%)
sounds frustrating, totally normal,great ways, on [your] plate,be overwhelming, sometimes feelfrightening, on top [of]been struggling, feeling alone
You have a lot of things on your plate, between familyand financial problems. [reflection]
It’s totally normal to feel lonely when you haveno one to talk to. [reflection]
Those are great ways to improve the relationship. [affirmation]
(middle 25%)
happened [to] make,mean [when you] say,is that, you recognized, source ofthe moment, are brave
Has anything happened to make you anxious? [exploration]It’s good you recognized the need to reach out. [affirmation]Can you tell me what you mean when you say
you’re giving up? [risk assessment]
Forwards-oriented(top 25%)
plan for, confided [to] anyone,usually do, has helped,been talking, best supporthave considered, any activities
Can you think of anything that has helped whenyou’ve been stressed before? [problem solving]
I want to be the best support for you today. [problem solving]We’ve been talking for a while now, how do you feel? [closing]
Table 1: Example phrasings and sentences with labeled strategies from crisis counselors’ messages, at varyingorientations: backwards-oriented (from the bottom 25% of Ω), middle, and forwards-oriented (from top 25%).
A
B
Figure 4: Validating the orientation measure and com-paring to alternatives. A Leftmost: Mean Ω per coun-seling strategy label (vertical line denotes Ω=0). Nextthree: same for other measures. B: Mean Ωmax andΩmin per segment for risk-assessed (orange) and non-risk-assessed (black) conversations. Both: Solid circlesindicate statistically significant differences (Wilcoxonp<0.01, comparing within-counselor).
5.2 Validation: Conversation Structure
We also show that orientation tracks with thestructure of crisis counseling conversations as de-scribed in the training material. Following Althoffet al. (2016), we divide each conversation with atleast ten counselor messages into five equal-sizedsegments and average Ωmax and Ωmin over mes-sages in each segment.
Figure 4B (black lines) shows that over thecourse of a conversation, messages tend to getmore forwards-oriented (higher Ωmax and Ωmin).This matches a standard conversation structure
taught in the training: addressing the texter’s exist-ing problems before advancing towards problem-solving. While this correspondence holds in ag-gregate, orientation also captures complementaryinformation to advancement through stages—e.g.,while problem-solving, counselors may still ad-dress and affirm a texter’s ideas (Table 1, row 3).
We also consider a subset of conversationswhere we expect a different trajectory: for po-tentially suicidal texters, the training directs coun-selors to immediately start a process of risk as-sessment in which actively prompting the texterto disclose their level of suicidal ideation takesprecedence over other objectives. As such, weexpect more forwards-oriented messages at thestarts of conversations involving such texters. In-deed, in the 30% of conversations which are risk-assessed, we find significantly larger Ωmax in thefirst segment (Figure 4B, orange line; Wilcoxonp < 0.01 in the first stage, comparing within-counselor). Ωmin is smaller at each stage, sug-gesting that counselors balance actively promptingthese critical disclosures with addressing them.
5.3 Alternative Operationalizations
We compare orientation to other methods for cap-turing a counselor’s balancing decisions:Naive distance. We conside the naive approach inSection 4, taking a difference in cosine distancesbetween tf-idf representations of a message and itsreply, and a message and its predecessor.Backwards-range. We consider just the mes-sage’s backwards-range. For each sentence wetake tf-idf weighted averages of component ←−σwand take minimum←−σ for each message.7
7We get qualitatively similar results with maximum −→σ .
Question-asking. We consider whether the mes-sage has a question. This was used in Walkerand Whittaker (1990) as a signal of taking control,which could be construed as forwards-oriented.
Within-label standard deviations of each al-ternative measure are generally not significantlysmaller than across-label (Figure 4A), indicatingthat these measures are poorer reflections of thecounseling strategies. Label rankings under themeasures are also arguably less intuitive. For in-stance, reflection statements have relatively large(naive) cosine distance from their predecessors. In-deed, the training encourages counselors to pro-cess rather than simply restate the texter’s words.
These measures also track with the conversa-tion’s progress differently—notably, none of themdistinguish the initial dynamics of risk-assessedconversations as reflected in Ωmax (see appendix).
5.4 Relation to Conversational Effectiveness
Past work on counseling has extensively dis-cussed the virtues of addressing a client’s situation(Rogers, 1957; Hill and Nakayama, 2000). Somestudies also suggest that accounting for both ad-dressing and advancing is important, such that ef-fective counselors manage to mix backwards- andforwards-oriented actions (Mishara et al., 2007).
We use orientation to examine how these strate-gies are tied to conversational effectiveness in cri-sis counseling at a larger scale, using our measuresto provide a unified view of advancing and address-ing. To derive simple conversation-level measures,we average Ωmax and Ωmin over each counselormessage in a conversation.
Adjudicating counseling conversation quality isknown to be difficult (Tracey et al., 2014). Asa starting point, we relate our conversation-levelmeasures to two complementary indicators of aconversation’s effectiveness:8
Perceived helpfulness. We consider responsesfrom a post-conversation survey asking the texterwhether the conversation was helpful, followingAlthoff et al. (2016). Out of the 26% of conversa-tions with a response, 89% were rated as helpful.9
Conversation length. We consider a conversation’slength as a simple indicator of the pace of itsprogress: short conversations may rush the tex-ter, while prolonged conversations could suggest
8We perform all subsequent analyses on a subset of234,433 conversations, detailed in the appendix.
9We note that this indicator is limited by important factorssuch as the selection bias in respondents.
helpful
unhelpful
more forwards-orientedmore backwards-oriented
mor
e fo
rwar
ds-
orie
nted
mor
e ba
ckw
ards
-or
ient
ed
A
B
C
Figure 5: Relation between orientation and conversa-tional effectiveness. A: Mean Ωmin and Ωmax in con-versations rated as helpful (green) or unhelpful (grey)(macroaveraged per conversation). Differences in bothmeasures are significant (Mann Whitney U test p <0.001). B, C: Mean Ωmin and Ωmax of conversationswith varying lengths (in # of messages). Both plots: Er-ror bars show 95% bootstrapped confidence intervals.
stalling and could even demoralize the counselor(Landphair and Preddy, 2012).10
Figure 5A compares Ωmin and Ωmax in con-versations rated as helpful and unhelpful by tex-ters. Both measures are significantly smallerin conversations perceived as helpful, suggestingthat texters have a better impression of relativelybackwards-oriented interactions where the coun-selor is inclined towards addressing their situation.As such, this result echoes past findings relatingaddressing to effectiveness.
Figure 5B compares Ωmin in conversations ofvarying lengths, showing that Ωmin increases withlength, such that counselors exhibit less propensityfor addressing in longer conversations. Anecdo-tal observations cited in interviews with the plat-form’s staff suggest one interpretation: conversa-tions in which a texter feels their concerns werenot satisfactorily addressed may be prolongedwhen they circle back to revisit these concerns.
Figure 5C relates Ωmax to conversation length.We find that Ωmax is smaller in the lengthiestconversations, suggesting that such prolonged in-
10As the training material notes, conversation length andtexter perception may signal complementary or even conflict-ing information about a texter’s experience of a conversationand its effectiveness: “Some texters resist the end of the con-versation. They ruminate [...] causing the conversation todrag on without any progress.”
teractions may be stalled by a weaker impulseto advance forwards. Extremely short conversa-tions have smaller Ωmax as well, such that pre-mature endings may also reflect issues in advanc-ing. As such, we add credence to the previously-posited benefits of mixing addressing and advanc-ing: forwards-oriented actions may be tied to mak-ing progress, while a weaker propensity to ad-vance may signal a suboptimal pace.Counselor-level analysis. These findings couldreflect various confounds—for instance, a coun-selor’s choice of orientation may have no bearingon the rating they receive from a particularly dif-ficult texter. To address this, we compute simi-lar correspondences between orientation and oureffectiveness indicators at the level of counselorsrather than conversations; this analysis is detailedin the appendix. Our conversation-level results arereplicated under these controls.
6 Discussion and Future Work
In this work, we sought to examine a key bal-ance in crisis counseling conversations betweenadvancing forwards and addressing what has al-ready been said. Realizing this balance is one ofthe many challenges that crisis counselors mustmanage, and modeling the actions they take inlight of such challenges could point to policiesto better support them. For instance, our methodcould assist human supervisors in monitoring theprogress of ongoing conversations to detect in-stances of rushing or stalling, or enable larger-scale analyses of conversational behaviors to in-form how counselors are trained. The unsuper-vised approach we propose could circumvent dif-ficulties in getting large-scale annotations of suchsensitive content.
Future work could bolster the measure’s useful-ness in several ways. Technical improvements likericher utterance representations could improve themeasure’s fidelity; more sophisticated analysescould better capture the dynamic ways in whichthe balance of objectives is negotiated across manyturns. The preliminary explorations in Section 5.4could also be extended to gauge the causal effectsof counselors’ behaviors (Kazdin, 2007).
We expect balancing problems to recur in con-versational settings beyond crisis counseling, suchas court proceedings, interviews, debates and othermental health contexts like long-term therapy. Inthese settings, individuals also make potentially
consequential choices that span the backwards-forwards orientation axis, such as addressing pre-vious arguments (Tan et al., 2016; Zhang et al.,2016) or asking leading questions (Leech, 2002).Our measure is designed to be broadly applica-ble, requiring no domain-specific annotations; weprovide exploratory output on justice utterancesfrom the Supreme Court’s oral arguments in theappendix and release code implementing our ap-proach at http://convokit.cornell.edu to en-courage experiments in other domains. However,the method’s efficacy in the present setting is likelyboosted by the relative uniformity of crisis coun-seling conversations; and future work could aimto better accomodate settings with less structureand more linguistic variability. With such improve-ments, it would be interesting to study other do-mains where interlocutors are faced with conver-sational challenges.Acknowledgements. We thank Jonathan P.Chang, Caleb Chiam, Liye Fu, Dan Jurafsky, JackHessel, and Lillian Lee for helpful conversations,and the anonymous reviewers for their thoughtfulcomments. We also thank Ana Smith for collect-ing and processing the Supreme Court oral argu-ment transcripts we used in the supplementary ma-terial. This research, and the counseling serviceexamined herein, would not have been possiblewithout Crisis Text Line. We are particularly grate-ful to Robert Filbin, Christine Morrison, and Ja-clyn Weiser for their valuable insights into the ex-periences of counselors and for their help with us-ing the data. The research has been supported inpart by NSF CAREER Award IIS1750615 and aMicrosoft Research PhD Fellowship. The collabo-ration with Crisis Text Line was supported by theRobert Wood Johnson Foundation; the views ex-pressed here do not necessarily reflect the viewsof the foundation.
References
Tim Althoff, Kevin Clark, and Jure Leskovec. 2016.Large-scale Analysis of Counseling Conversations:An Application of Natural Language Processing toMental Health. Transactions of the Association forComputational Linguistics.
David C. Atkins, Mark Steyvers, Zac E. Imel, andPadhraic Smyth. 2014. Scaling up the evaluation ofpsychotherapy: Evaluating motivational interview-ing fidelity via statistical text classification. Imple-mentation Science.
Graham D. Bodie, Andrea J. Vickery, Kaitlin Cannava,and Susanne M. Jones. 2015. The Role of “ActiveListening” in Informal Helping Conversations: Im-pact on Perceptions of Listener Helpfulness, Sensi-tivity, and Supportiveness and Discloser EmotionalImprovement. Western Journal of Communication.
David Bracewell, Marc Tomlinson, and Hui Wang.2012. Identification of Social Acts in Dialogue. InProceedings of COLING.
Dogan Can, David C. Atkins, and Shrikanth S.Narayanan. 2015. A dialog act tagging approachto behavioral coding: A case study of addictioncounseling conversations. In Proceedings of INTER-SPEECH.
Jie Cao, Michael Tanana, Zac Imel, Eric Poitras, DavidAtkins, and Vivek Srikumar. 2019. Observing Dia-logue in Therapy: Categorizing and Forecasting Be-havioral Codes. In Proceedings of ACL.
Jonathan P Chang, Caleb Chiam, Liye Fu, An-drew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for theAnalysis of Conversations. In Proceedings of SIG-DIAL.
Mark G Core and James F Allen. 1997. Coding Di-alogs with the DAMSL Annotation Scheme. AAAIfall symposium on communicative action in humansand machines.
Jacques Gaume, Nicolas Bertholet, Mohamed Faouzi,Gerhard Gmel, and Jean-Bernard Daeppen. 2010.Counselor motivational interviewing skills andyoung adult change talk articulation during brief mo-tivational interventions. Journal of Substance AbuseTreatment.
Barbara J. Grosz, Aravind K. Joshi, and Scott Wein-stein. 1995. Centering: A Framework for Modelingthe Local Coherence of Discourse. ComputationalLinguistics.
Clara E. Hill and Emilie Y. Nakayama. 2000. Client-centered therapy: Where has it been and where is itgoing? A comment on Hathaway (1948). Journal ofClinical Psychology.
Jon Houck. 2008. Motivational Interviewing SkillCode (MISC) 2.1.
Neil P. Jones, Alison A. Papadakis, Caitlin M. Hogan,and Timothy J. Strauman. 2009. Over and overagain: Rumination, reflection, and promotion goalfailure and their interactive effects on depressivesymptoms. Behaviour Research and Therapy.
Alan E. Kazdin. 2007. Mediators and mechanisms ofchange in psychotherapy research. Annual Reviewof Clinical Psychology.
Juliette Landphair and Teri Preddy. 2012. More thantalk: Co-Rumination among college students. AboutCampus.
Fei-Tzin Lee, Derrick Hull, Jacob Levine, Bonnie Ray,and Kathy McKeown. 2019. Identifying therapistconversational actions across diverse psychothera-peutic approaches. In Proceedings of the Sixth Work-shop on Computational Linguistics and Clinical Psy-chology.
Beth L. Leech. 2002. Asking Questions: Techniquesfor Semistructured Interviews. Political Science &Politics.
William C. Mann and Sandra A. Thompson. 1988.Rhetorical Structure Theory: Toward a functionaltheory of text organization. Text - InterdisciplinaryJournal for the Study of Discourse.
Brian L. Mishara, Francois Chagnon, Marc S. Daigle,Bogdan Balan, Sylvaine Raymond, Isabelle Mar-coux, Cecile Bardon, Julie K. Campbell, and Alan D.Berman. 2007. Which helper behaviors and inter-vention styles are related to better short-term out-comes in telephone crisis intervention? Results froma Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network. Suicide & life-threatening be-havior.
Johanna D. Moore and Cecile Paris. 1993. PlanningText for Advisory Dialogues: Capturing Intentionaland Rhetorical Information. Computational Linguis-tics.
Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik,Deborah A. Cai, Jennifer E. Midberry, and YuanxinWang. 2014. Modeling topic control to detect in-fluence in conversations using nonparametric topicmodels. Machine Learning.
Susan Nolen-Hoeksema, Blair E. Wisco, and SonjaLyubomirsky. 2008. Rethinking Rumination. Per-spectives on Psychological Science.
Sungjoon Park, Donghyun Kim, and Alice Oh. 2019.Conversation Model Fine-Tuning for ClassifyingClient Utterances in Counseling Dialogues. In Pro-ceedings of NAACL.
Veronica Perez-Rosas, Rada Mihalcea, Kenneth Resni-cow, Satinder Singh, and Lawrence An. 2017. Un-derstanding and Predicting Empathic Behavior inCounseling Therapy. In Proceedings of ACL.
Veronica Perez-Rosas, Xuetong Sun, Christy Li,Yuchen Wang, Kenneth Resnicow, and Rada Mihal-cea. 2018. Analyzing the Quality of CounselingConversations: The Tell-Tale Signs of High-qualityCounseling. In Proceedings of LREC.
Anthony R. Pisani, Nitya Kanuri, Bob Filbin, CarlosGallo, Madelyn Gould, Lisa S. Lehmann, RobertLevine, John E. Marcotte, Brian Pascal, DavidRousseau, Shairi Turner, Shirley Yen, and Megan L.Ranney. 2019. Protecting User Privacy and Rightsin Academic Data-Sharing Partnerships: PrinciplesFrom a Pilot Program at Crisis Text Line. Journalof Medical Internet Research.
Vinodkumar Prabhakaran, Camilla Griffiths, Hang Su,Prateek Verma, Nelson Morgan, Jennifer L. Eber-hardt, and Dan Jurafsky. 2018. Detecting Institu-tional Dialog Acts in Police Traffic Stops. Transac-tions of the Association for Computational Linguis-tics.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Un-supervised Modeling of Twitter Conversations. InProceedings of NAACL.
Carl R. Rogers. 1957. The necessary and sufficient con-ditions of therapeutic personality change. Journal ofConsulting Psychology.
Stephen Rollnick and William R. Miller. 1995. What isMotivational Interviewing? Behavioural and Cogni-tive Psychotherapy.
Amanda J. Rose, Wendy Carlson, and Erika M.Waller. 2007. Prospective Associations of Co-Rumination with Friendship and Emotional Adjust-ment: Considering the Socioemotional Trade-Offsof Co-Rumination. Developmental Psychology.
Sara Rosenthal and Kathleen McKeown. 2015. ICouldn’t Agree More: The Role of ConversationalStructure in Agreement and Disagreement Detectionin Online Discussions. In Proceedings of SIGDIAL.
Harvey Sacks. 1992. Lectures on Conversation. Black-well.
Jonathan Sandoval, Amy Nicole Scott, and IrenePadilla. 2009. Crisis counseling: An overview. Psy-chology in the Schools.
Emanuel A. Schegloff. 1987. Some sources of misun-derstanding in talk-in-interaction. Linguistics.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu, and Lillian Lee. 2016. Winning Argu-ments: Interaction Dynamics and Persuasion Strate-gies in Good-faith Online Discussions. In Proceed-ings of WWW.
Michael Tanana, Kevin A. Hallgren, Zac E. Imel,David C. Atkins, and Vivek Srikumar. 2016. A Com-parison of Natural Language Processing Methodsfor Automated Coding of Motivational Interviewing.Journal of Substance Abuse Treatment.
Jenny Thomas. 1983. Cross-cultural pragmatic failure.Applied Linguistics.
Terence J. G. Tracey, Bruce E. Wampold, James W.Lichtenberg, and Rodney K. Goodyear. 2014. Ex-pertise in psychotherapy: An elusive goal? TheAmerican Psychologist.
Marilyn Walker and Steve Whittaker. 1990. Mixed Ini-tiative in Dialogue: An Investigation into DiscourseSegmentation. In Proceedings of ACL.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh,Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Per-suasion for Good: Towards a Personalized Persua-sive Dialogue System for Social Good. In Proceed-ings of ACL.
Bonnie Lynn Webber. 2001. Computational Perspec-tives on Discourse and Dialog. In The Handbook ofDiscourse Analysis. John Wiley & Sons, Ltd.
Harry Weger, Gina R. Castle, and Melissa C. Emmett.2010. Active Listening in Peer Interviews: The In-fluence of Message Paraphrasing on Perceptions ofListening Skill. International Journal of Listening.
Justine Zhang, Robert Filbin, Christine Morrison, Ja-clyn Weiser, and Cristian Danescu-Niculescu-Mizil.2019. Finding Your Voice: The Linguistic Develop-ment of Mental Health Counselors. In Proceedingsof ACL.
Justine Zhang, Ravi Kumar, Sujith Ravi, and CristianDanescu-Niculescu-Mizil. 2016. ConversationalFlow in Oxford-style Debates. In Proceedings ofNAACL.
Justine Zhang, Arthur Spirling, and Cristian Danescu-Niculescu-Mizil. 2017. Asking too Much? TheRhetorical Role of Questions in Political Discourse.In Proceedings of EMNLP.
A Appendices
A.1 Further Details About Methodology
Here, we provide further details on our methodol-ogy for measuring orientation, to supplement thedescription in Section 4.2 and aid reproducibility.
Our aim in the first part of our methodologyis to measure the orientation of phrasings Ωw.We would like to ensure that our measure is notskewed by the relative frequencies of phrasings,and take two steps to this ends, which empiri-cally produced more interpretable output. First,we scale rows of term-document matrix X (corre-sponding to texter phrasings) to unit ℓ2 norm be-fore deriving their representation in T via singularvalue decomposition. Second, we remove the firstSVD dimension, i.e., first column of U , and renor-malize each row, before proceeding.
A.2 Further Details About Application toCounseling Data
Here, we provide further details on how we ap-plied our methodology to the dataset of counsel-ing conversations in order to measure the orienta-tion of counselor utterances, as described in Sec-tion 5. In particular, we list empirical choices
0% 20% 40% 60% 80%conversation segment
0.03
0.02
0.01naive distance
0% 20% 40% 60% 80%conversation segment
0.810
0.815
0% 20% 40% 60% 80%conversation segment
40
60
% ?s
Figure 6: Mean naive distance, backwards-range (←−σ ), and % of utterances with questions, per segment for risk-assessed (orange) and non-risk-assessed (black) conversations; solid circles indicate statistically significant differ-ences (Wilcoxon p < 0.01, comparing conversation types within counselor).
made in extracting and then processing the train-ing set of 351,935 texter and counselor messagesused to measure phrasing orientations.
We randomly sampled 20% of counselors in thedata; all conversations by these counselors wereomitted in subsequent analyses. We merged con-secutive messages from the same interlocutor. Tomitigate potential noise in characterizing phras-ings, we considered only counselor and texter mes-sage pairs in which each message has between 15and 45 words. We extracted all messages from theconversations which met these constraints.
We represent counselor phrasings asdependency-parse arcs and texter messagesas unigrams, reflecting the comparatively struc-tured language of the counselors versus the texters(counselors are instructed to speak in grammat-ically well-formed sentences). We consider the5000 most frequent phrasings for each role, anddiscard sentences without any such phrasings.Finally, we used 25 SVD dimensions to induce T.
A.3 Full Listing of Counselor Action Labels
Table 2 lists each counseling action derived fromthe training material and used during the valida-tion procedure (Section 5.1) to label sentences.
A.4 Orientation and Lexical Properties
Here, we supplement our discussion of simple lex-ical properties that could be used to characterizemessages (Section 5.3), discussing how orienta-tion reflects these properties and showing that ori-entation is not subsumed by them.Backwards-range. As seen in their weakbackwards-range (high, i.e., spread-out ←−σ ), affir-mations that highlight the texter’s strengths canfollow a variety of situations. However, the repliesthey prompt are yet more diffuse, emphasizing theneed to compare both directions.
reflection (113)re-wording to show understanding and validate feelingsIt can be overwhelming to go through that on your own.affirmation (60)pointing out the texter’s positive qualities and actionsYou showed a lot of strength in reaching out to us.exploration (44)prompting texters to expand on their situationIs this the first real fight you’ve had with your boyfriend?problem solving (110)identifying the texter’s goals and potential coping skillsWhat do you usually do to help you feel calmer?closing (43)reviewing the conversation and transitioning to a closeI think you have a good plan to get some rest tonight.risk assessment (19)assessing suicidal ideation or risk of self-harmDo you have access to the pills right now?
Table 2: Counseling strategies and representative exam-ples derived from the training material. The number ofsentences (out of 400) assigned to each label is shownin parentheses (11 were not labeled as any action).
Question-asking. We see that questions—whichnominally prompt the texter for a response—aremore forwards-oriented than non-questions; 61%of sentences with ‘?’ have Ω > 0 comparedto 21% of sentences without. However, thesenumbers also show that explicitly-marked ques-tions are inexact proxies of forwards-orientedsentences—as in Table 1, questions can addressa past remark by prompting clarifications, whilecounselors can use non-questions to suggest anintent to advance stages (e.g., to transition toproblem-solving).
A.5 Relating Alternate Measures toConversation Progress
Figure 6 shows averages per conversation segment(i.e., 20% of a conversation) for each alternativemeasure considered in Section 5.3. Comparing tothe average Ωmax and Ωmin shown in Figure 4, wesee that these measures track with the conversa-
tion’s progress differently, and none of them distin-guish the initial dynamics of risk-assessed conver-sations as dramatically as reflected in Ωmax, e.g.,simple counts of questions do not distinguish be-tween questions geared towards risk-assessmentversus more open-ended problem exploration.
A.6 Further Details About Data Used inAnalyses
Here, we provide further details about the subsetof data we used to analyze counselors’ orienta-tion behavior (Section 5.4). In particular, our aimwas to characterize behavior in typical conversa-tions rather than exceptional cases or those thatreflected earlier versions of the training curricu-lum. As such, we only considered the 234,433conversations which had least five counselor mes-sages, were not risk-assessed or disconnected be-fore completion, and were taken by counselorswho joined the platform after January 2017.
A.7 Counselor-Level AnalysisHere, we provide further details about our pro-cedure for analyzing counselor-level correspon-dences between orientation and effectiveness indi-cators, as alluded to in Section 5.4.
Recall that our conversation-level findings maybe confounded by texter idiosyncracies: for in-stance, texters with particularly difficult situationsmight affect a counselor’s behaviour, but may alsobe more likely to give bad ratings, independentof how the counselor behaves. Alternatively, anoverly long conversation could arise because thecounselor is less forwards-oriented, or because thetexter is reluctant to make progress from the out-set, making it hard for the counselor to attempt toprompt them forwards.
To separate a counselor’s decisions from thesesituational effects, we take a counselor-level per-spective. While counselors cannot selectivelytalk with different types of texters, they canexhibit cross-conversational inclinations for par-ticular behaviors. We therefore relate thesecross-conversational tendencies in orienting a con-versation to a counselor’s long-term propensityfor receiving helpful ratings, or having long orshort conversations. We proceed to describe ourmethodology for relating counselor tendencies toperceived helpfulness; an analogous procedurecould be applied to conversation length as well.
We characterize a counselor’s orienting behav-ior as the average Ωmax and Ωmin over the con-
versations they take; we likewise take the pro-portion of their (rated) conversations which wereperceived as helpful. We restrict our counselorlevel analyses to the 20th to 120th conversationsof the 1495 counselors who have taken at least 120conversations (ignoring their initial conversationswhen they are still acclimatizing to the platform).
To cleanly disentangle counselor tendency andconversational circumstance, we split each coun-selor’s conversations into two interleaved subsets(i.e., first, third, fifth . . . versus second, fourth. . . conversations), measuring orientation on onesubset and computing a counselor’s propensity forhelpful ratings on the other. Here, we draw an anal-ogy to the machine learning paradigm of taking atrain-test split: “training” counselor tendencies onone subset and “testing” their relation to rating onthe other subset. In general, the directions of theeffects we observe hold with stronger effects if wedo not take this split.
Echoing conversation-level effects, counselorsthat tend to be less forwards-oriented and morebackwards-oriented (those in the bottom thirds ofΩmax and Ωmin respectively) are more likely to beperceived as helpful; this contrast is stronger interms of Ωmin (Cohen’s d = 0.30, p < 0.001)than Ωmax (d = 0.13, p < 0.05), suggesting thata counselor’s tendency for advancing weighs lesson their perceived helpfulness than their tendencyfor addressing. Also in line with the conversation-level findings, counselors with smaller Ωmax tendto have longer conversations (d = 0.54, p <0.001), as do counselors with larger Ωmin (d =0.17)—here, a counselor’s tendency for advancingis more related to their propensity for shorter con-versations than their tendency for addressing.
We note that counselors on the platform cannotselectively take conversations with certain texters;rather, the platform automatically assigns incom-ing texters to a counselor. As such, the counselor-level effects we observe cannot be explained bycounselor self-selection for particular situations.
A.8 Orientation in Multi-SentenceUtterances
Our motivation in characterizing utterances us-ing the minimum and maximum sentence orien-tation was to reflect potential heterogeneities inutterances which could be both backwards- andforwards-oriented (consider a message where c2and c1 from Figure 1 are concatenated). Examin-
Orientation Example phrasings Example sentences
Less forwards-oriented(bottom 25%)
i understand, have been,part of, so you,sentence, talking aboutmight, particularbut the, give to
As I understand the facts [...] he had tried to kill the husband,shooting him twice in the head? (Scalia)
You started out by talking about what the first jury knew,but [...] we aren’t reviewing that determination. (Roberts)
I guess the problem is the list of absurdities that they point to,not the least of which is a dry dock. (Sotomayor)
So you hedged, because it’s very hard to find the right sentence. (Breyer)
More forwards-oriented(top 25%)
hypothetical, would have,agree, difference [between],[do] you think, your positionyour argument, a questionapply, was there
Suppose under this hypothetical [...] the judge doesn’t sayaggravated murder when he submits it to the jury. (Kennedy)
I just want to know your position on the second, the cart before thehorse point. (Souter)
Do you also agree [...] that if not properly administered there is somerisk of excruciating pain? (Stevens)
What’s the difference between pigment and color [...] ? (Ginsburg)
Table 3: Example phrasings and sentences from utterances of Supreme Court justices, identified in parentheses,which are less or more forwards-oriented (bottom and top 25% of Ω).
ing the 64% of counselor messages with multiplesentences, we see that 52% of these messages haveΩmin < 0 and Ωmax > 0. Our method, which isable to account for this heterogeneity, thus pointsto one potential strategy for counselors to bridgebetween both objectives.
A.9 Application to Supreme Court OralArguments
Here, we include an exploratory study of how ourapproach could be adapted to analyze domains be-yond crisis counseling conversations, as alluded toin Section 6. We apply the method to measurethe orientation of utterances by Supreme Court jus-tices during oral arguments, when they engage inexchanges with lawyers (so justices and lawyersplay the roles of counselor and texter, respectively,in our method). We used transcripts of 668 cases,taken from the Oyez project (https://www.oyez.org/), averaging 120 justice utterances per case.11
Oral arguments contain more linguistic and top-ical heterogeneiety than counseling conversations,since they cover a wide variety of cases, and be-cause the language used by each justice is moredifferentiated. In addition, the dataset is muchsmaller. As such, this represents a more challeng-ing setting than the counseling context, requiringchanges to the precise procedure used to measureorientation, and pointing to the need for furthertechnical improvements, discussed in Section 6.
Nonetheless, our present methodology is able toproduce interpretable output. Table 3 shows rep-resentative phrasings and (paraphrased) sentenceswith different orientations. In contrast to the coun-
11The data used can be found at http://analysmith.com/research/scotus/data.
seling domain, 70% of phrasings and 93% of sen-tences have Ω > 0, perhaps reflecting the partic-ular power dynamic in the Supreme Court, wherejustices are tasked with scrutinizing the argumentsmade by lawyers. We find that highly forwards-oriented phrasings and utterances tend to reflectjustices pressing on the lawyers to address a point(e.g., do you agree, what’s the difference between);the least forwards-oriented phrases involve the jus-tice rehashing and reframing (not always in com-plimentary terms) a lawyer’s prior utterances (e.g.,so you [...], [as] i understand).
We used a training set of 15,862 justice andlawyer messages, where each utterance had be-tween 10 and 100 words. Both lawyer and justiceutterances were represented as dependency-parsearcs. Empirically, we found that the methodologywas sensitive to idiosyncracies of particular casesand justices. To minimize this effect, we restrictedthe size of the justice’s vocabulary by only consid-ering the 398 justice phrasings which occurred inat least 200 utterances.