activity highlights dd - cis.jhu.edu · 0,stellax,paper_follow_edit,important_person,ml for...

Activity HighlightsXugang

Overview

• Product level

• Surface most relevant activities from collaborators

• Display reason(s) for the highlights

• Technical level

• Build the activity ranker(s)

• Analyze the relevance/importance

Similar problems

• Emails

• Surface most relevant emails from contacts

• Display snippets

• Posts

• Surface most relevant posts from friends

• Display thumbnails and comments

Formulation

• Activity representation

(� = �, �, �, � , �)

where

� is doc, which can be accessed by the user

�is person, the one who had the activity

�is activity type, which is one of the pre-defined actions

� is timestamp, the time when the activity occurred

� is the user

Formulation

• Importance probability

(�|�) = �, �, �, �|�

• Factorization

�, �, �, �|� = � �, �, �, � (�|�, �, �)(�|�, �) � �

• Approximation

�, �, �, �|� ≈ � �, � (�|�, �)(�|�) � �

• Ranking score

� �, � ≈ log � �, � + log � �, � + log � � + log �|�

Formulation

• Per-person ranking

�|�, � = �, �, � �, � ≈ �|�, � �|�, � (�|�) �|� /(�|�)

� �|�, � = � �, � − log (�|�)

• Per-doc ranking

�|�, � = (�, �, �|�, �) ≈ �|�, � �|�, � �|�

� �|�, � = � �, � − log (�|�)

Distribution assumptions

• Recency

�|�~exponential

• Doc importance

��(��,�)

= � � , ��(��,�)~ Dirichlet �� ,�

• Per-doc person importance:

��(�� !�",��,�)

= �|�, � , ��(�� !�",��,�)~ Dirichlet �� !�",��,�

• Per-doc activity importance:

�#($�%#&#%',��,�)

= �|�, � , ��($�%#&#%',��,�)~ Dirichlet �� $�%#&#%',��,�

• Person importance

��(�� !�",�)

= � � , ��(�� !�",�)~ Dirichlet �� !�",�

Distributions

• Exponential

� = exp −+ � �, − � , + > 0, � �, > �

• Dirichlet

�� = Dirichlet �� ,

/ �� = ��/�, � = ∑ �11 ,

��|2 = Dirichlet �� + 2 ,

/ ��|2 = �� + 2 / � + 2 = �/ �� + 2 / � + 2 , 2 = ∑ 211

Sliding window estimation

Sliding window

Labeling

window�3

Labeling

window�4

Time5 5 + ∆5

7 �3|�, 5 7 �4|�, 5

Validation window

8 9

9 9

Basic ideas:

• (�|�) is estimated by counting the user’s

activities on the doc. That is

� � = ∑ � �(�,�), � �(�,�) �%(:,;)

= ∑ �(�,�) �%(:,;) .

• Each activity by “collaborator” is labeled by

counting the user’s activities that follow on

the same doc.

• The labels are used , with smoothing, to

estimate �|�, � , �|�, � , and (�|�).

Sliding window evaluation

Sliding window

Labeling

window�3

Labeling

window�4

Time5 5 + ∆5

7 �3|�, 5 7 �4|�, 5

Validation window

8 9

9 9

Basic ideas:

• When each activity in the sliding window is

labeled by counting the user’s following

activities on the same doc in the validation

window, the “gain” at each position of the

ordered list can be evaluated

• Once the gain at each position of the ordered

list is obtained, some metrics like NDCG and

recall can be calculated.

Evaluation metrics

Suppose from a sliding window [5 − 8, 5], a list of ordered activities �3, … , �"? are generated, with ranking scores � @AB,@ (�3) ≥ ⋯ ≥ � @AB,@ (�"?). Suppose for E = 1,… , 2@, the activity �1 has the count G1 (as label) of the user’s following activities on the same doc.

• NDCG

2�GH �3, … , �"? = ∑�I

JKLM(1N3)"1O3 / ∑

�I∗

JKLM(1N3)"1O3 ,

where G3∗ ≥ ⋯ ≥ G"?

∗ is the reorder of G3, … , G"?.

• Recall

QRG�99S �3, … , �"? = ∑ G1S1O3 / ∑ G1

"1O3 ,

where T ≤ 2 is the cut-off positon.

Suppose �3, … , �"? are reordered in the validation window [5 − 8, 5 + 9] into �3∗ , … , �"?

∗ , with labels G3∗ ≥

⋯ ≥ G"?∗ , respectively.

• Rank-order reward

Q �3, … , �"? ="("N3)

4∑

"A1N3

JKLM(|VIA1|N4)"1O3 ,

where W1 is the position of �1 in �3∗ , … , �"?

∗

Justification heuristics

• Doc importance(�|�)

• Cumulativedoc importance

` � � = ∑ (�V|�)V1O3 , where �3 � ≥ ⋯ ≥ �a: � , �V = �

• Person importance(�|�),

• Cumulative person importance

` � � = ∑ (�V|�)V1O3 , where �3 � ≥ ⋯ ≥ �ab � , �V = �

• Recency(�|�)

Explanation

• Compare (�|�) and (�|�)

• Compare ` � � , ` � � with respective thresholds

• Compare ` � � , ` � � with respective thresholds, and set position cut-off (e.g., 5) in rank order (the one ever used by MS SharePoint)

• New event(s) ((�|�)) on relevant docs ((�|�) > 0)

Grouping

�3, �3, �3, �3�3, �3, �3, �4�3, �4, �3, �c�4, �4, �4, �d�3, �4, �3, �e�3, �3, �3, �f�3, �c, �c, �g�4, �c, �4, �h�4, �d, �3, �i�3, �d, �d, �3j�c, �e, �3, �33�d, �e, �4, �34

�3, �3, �3,·

�3, �4, �3,·

�4, �4, �4,·

�3, �c, �c, �g

�4, �c, �4, �h

�4, �d, �3, �i

�3, �d, �d, �3j

�c, �e, �3, �33

�d, �e, �4, �34

�3, (�3, �4), �3,·

�4, (�4, �c), �4,·

�3, �c, �c, �g�4, �d, �3, �i�3, �d, �d, �3j�c, �e, �3, �33�d, �e, �4, �34

Raw ranked list Group by �, �, � Group by �, �

For you

Suggested for you

Team highlights

• Compress the ranked list in presentation

Antenna

ImplementationUI:

Web Desktop Mobile

…Users

Files …

User activities

File activities

Event Store

(HBase)

Index

(HDFS)

Antenna requests

(user ID, time range)

User’s

activities

Other’s

activities

Basic schema:

object_type, action,

timestamp, ns_path,

unique_identifier,

user_idService requests

(user ID, reference time, model)

Ranked list

of other’s

activities

Model library

Probability/Statistics

Heuristics

Supervised Learning

Models

Metrics

Calculation

Metrics

reports

Postprocessing

(grouping)

Implementation

from dropbox.context import Context

from dropbox.ml_platform.suggest_backend.tools.activity_highlight_debugging import

run_as_user, annotate_events_and_score_results

results = activity_highlight_debugging.run_as_user(

Context.BACKGROUND,

user,

ns_ids,

UserActivityHighlightSource.QueryMode.QUERY_MORE_BUT_FETCH_LESS,

False,

heuristic_version)

annotations_and_scores = activity_highlight_debugging.annotate_events_and_score_results(

Context.BACKGROUND,

user_id,

results)

Implementation

from dropbox.ml_platform.suggest_backend.tools import activity_highlight_debugging

from dropbox.ml_platform.suggest_backend.experiments.sources.user_activity_sources import UserActivityHighlightSource

from dropbox.ml_platform.suggest_backend.tools.activity_highlight_debugging import _get_mock_request_logger

from dropbox.context import Context

from dropbox.iam.user import User

mode = UserActivityHighlightSource.QueryMode.QUERY_MORE_BUT_FETCH_LESS

activity_source = UserActivityHighlightSource(mode = mode, max_events=4500) # Instantiate

user_id = user_ids[0]

user = User.get(user_id)

ns_ids = activity_highlight_debugging._get_ns_ids_from_user(user)

logger = _get_mock_request_logger()

print "Getting user's and others' activities for " + "User {} ({})".format(user_id, user.email) + " ..."

# Query events in the type AntennaEvent

user_events_data_AntennaEvent, others_events_data_AntennaEvent = activity_source.get_activities(

Context.TODO(), # ctx, Context.BACKGROUND

user_id, # userid

ns_ids, # ns_ids

logger, # logger

UserActivityHighlightSource.QueryMode.QUERY_MORE_BUT_FETCH_LESS, # mode

None, # ref_time

None # actor_ids, set as none so that the method will get all the others who have acted in the ns

)

Implementation

from dropbox.vision.embeddings.activity_highlight.core import Event, create_event_from_antenna_event

user_events_data = [create_event_from_antenna_event(event)

for event in user_events_data_AntennaEvent]

others_events_data = [create_event_from_antenna_event(event)

for event in others_events_data_AntennaEvent]

Note:Event = NamedTuple('Event', [

('resource_type', antenna_proto.Object.ObjectType),

('resource_id', str),

('is_dir', Optional[bool]),

('actor_id', UserId),

('event_type', antenna_proto.ActionType),

('timestamp', Union[int, float]),

])

Implementation

from dropbox.vision.embeddings.activity_highlight.core import rank_activities

from dropbox.vision.embeddings.activity_highlight.core_v2 import rank_activities_v2

user_events_data_win = get_data_in_win(user_events_data, slidingWinStart_ts, slidingWinEnd_ts)

others_events_data_win = get_data_in_win(others_events_data, slidingWinStart_ts, slidingWinEnd_ts)

ranked_activities_v1 = rank_activities(

user_events_data_win,

others_events_data_win,

slidingWinEnd_ts,

variant)

ranked_activities_v2 = rank_activities_v2(

user_events_data_win,

others_events_data_win,

slidingWinEnd_ts,

variant)

Implementation

# type: (...) -> List[Tuple[Event, EventScoreBreakdown]]

EventScoreBreakdown = NamedTuple('EventScoreBreakdown', [

('overall_score', float),

('doc_score', float),

('actor_score', float),

('doc_rank', int),

('actor_rank', int),

])

Implementation

from dropbox.vision.embeddings.activity_highlight.core_v2 import label_others_events

user_events_data_valid = get_data_in_win(user_events_data, validationWinStart_ts, validationWinEnd_ts)

ranked_activities_v1_noscore = [event for event ,_ in ranked_activities_v1]

ranked_activities_v1_noscore_noVIEW = get_noVIEW_events(ranked_activities_v1_noscore)

ranked_activities_v2_noscore = [event for event ,_ in ranked_activities_v2]

ranked_activities_v2_noscore_noVIEW = get_noVIEW_events(ranked_activities_v2_noscore)

ranked_activities_v1_noscore_noVIEW_and_labels = label_others_events(user_events_data_valid,

ranked_activities_v1_noscore_noVIEW)

labels_v1 = [label for _, label in ranked_activities_v1_noscore_noVIEW_and_labels]

ranked_activities_v2_noscore_noVIEW_and_labels = label_others_events(user_events_data_valid,

ranked_activities_v2_noscore_noVIEW)

labels_v2 = [label for _, label in ranked_activities_v2_noscore_noVIEW_and_labels]

ndcg_v1 = calculate_ndcg(labels_v1)

ndcg_v2 = calculate_ndcg(labels_v2)

ndcgk_v1 = calculate_ndcg_cutoff(labels_v1, cutoff_position_ndcg)

ndcgk_v2 = calculate_ndcg_cutoff(labels_v2, cutoff_position_ndcg)

Implementation

from dropbox.vision.embeddings.activity_highlight.core_v2 import activities_people_doc_ranker

ranked_activities, people_weights_cweights, docs_weights_cweights

= activities_people_doc_ranker(user_events_data_win, others_events_data_win, slidingWinEnd_ts)

Results: snapshots of ranked lists of activitiesxugangye_v1.1.csv

,actor,event,reason,resource_name,ts0,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML for Activity Highlights Meeting Note,Fri Aug 2 21:21:19 2019

1,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML for Activity Highlights Meeting Note,Fri Aug 2 20:36:18 2019

2,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Aug 6 16:42:26 2019

3,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,ML for Activity Highlights Meeting Note,Fri Aug 2 20:36:18 2019

4,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,ML for Activity Highlights Meeting Note,Fri Aug 2 19:31:59 2019

5,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Aug 6 18:34:29 2019

6,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Jul 30 17:58:17 2019

7,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Activity Highlights Weekly Meeting,Mon Jul 29 19:17:41 2019

8,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,ML for Activity Highlights Meeting Note,Fri Jul 26 21:26:44 2019

9,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,ML for Activity Highlights Meeting Note,Fri Jul 26 17:39:10 2019

10,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Jul 30 20:25:23 2019

11,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Jul 30 16:59:25 2019

12,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Activity Highlights Weekly Meeting,Mon Jul 29 23:48:40 2019

13,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Labeled Dataset for Event Prediction,Tue Jul 30 21:44:33 2019



16,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Aug 6 18:34:29 2019

17,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Labeled Dataset for Event Prediction,Mon Jul 29 18:03:12 2019

18,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Labeled Dataset for Event Prediction,Mon Jul 29 17:30:43 2019

19,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Fri Aug 2 23:31:32 2019

20,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,D543845 Checklist,Thu Jul 25 01:25:16 2019


22,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Jul 30 19:37:11 2019

23,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Jul 30 05:46:40 2019

24,rkulkarni,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Workstream Q3 2019,Tue Jul 30 19:37:11 2019

25,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,AI Reading Group,Mon Jul 22 16:20:56 2019

26,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,AI Reading Group,Tue Aug 6 18:04:28 2019

27,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML Q2 2019 OKR Team Check Ins,Tue Jul 16 18:42:12 2019

28,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,AI Reading Group,Fri Aug 2 05:45:25 2019

29,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Q2 Retrospective,Mon Jul 8 18:44:39 2019

30,rkulkarni,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Onboarding for Ranjitha,Sun Jul 28 18:14:00 2019

31,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,AI Reading Group,Thu Jul 18 02:37:33 2019

32,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Embeddings Q2 Retrospective,Mon Jul 8 18:44:39 2019

33,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_RESOURCE,Allocating New Encrypted Machine,Mon Jul 15 20:46:38 2019

xugangye_v2.0.csv

,actor,event,reason,resource_name,ts0,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML for Activity Highlights Meeting Note,Fri Aug 2 20:36:18 2019

1,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML for Activity Highlights Meeting Note,Fri Aug 2 19:31:59 2019




5,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Activity Highlights Weekly Meeting,Mon Jul 29 23:48:40 2019

6,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Aug 6 18:34:29 2019

7,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Aug 6 18:34:29 2019

8,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Aug 6 16:42:26 2019


10,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Fri Aug 2 23:31:32 2019



13,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Jul 30 20:25:23 2019

14,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Jul 30 16:59:25 2019

15,stellax,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Workstream Q3 2019,Tue Jul 30 17:58:17 2019



18,rkulkarni,PAPER_FOLLOW_EDIT,UNSPECIFIED,Embeddings Workstream Q3 2019,Tue Jul 30 19:37:11 2019

19,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,AI Reading Group,Tue Aug 6 18:04:28 2019

20,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,AI Reading Group,Fri Aug 2 05:45:25 2019

21,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,AI Reading Group,Mon Jul 22 16:20:56 2019

22,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,ML Q2 2019 OKR Team Check Ins,Tue Jul 16 18:42:12 2019

23,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,AI Reading Group,Thu Jul 18 02:37:33 2019

24,ewei,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Q2 Retrospective,Mon Jul 8 18:44:39 2019

25,jongmin,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Embeddings Q2 Retrospective,Mon Jul 8 18:44:39 2019

26,xlyu,PAPER_FOLLOW_EDIT,IMPORTANT_PERSON,Allocating New Encrypted Machine,Mon Jul 15 20:46:38 2019

Results: snapshots of ranked lists of peopleRanked people list (top 30, for xugangye) from v2 in the sliding window

ending at 2019-08-05 00:10:49.468004:

user email|weight|cweight

tristan | 0.2092633745882235 | 0.2092633745882235

jongmin | 0.20283551430702448 | 0.412098888895248

stellax | 0.18603390264304223 | 0.5981327915382902

amelia | 0.08413206927665126 | 0.6822648608149415

chiyuanm | 0.03811628619853697 | 0.7203811470134786

xlyu | 0.026111685653104556 | 0.7464928326665831

edwardd | 0.02258914536214869 | 0.7690819780287318

ewei | 0.020352633694728176 | 0.78943461172346

agong | 0.01844836675084584 | 0.8078829784743058

clin | 0.01657514944744464 | 0.8244581279217504

ivant | 0.015282562850021999 | 0.8397406907717725

rickyk | 0.013058079867229348 | 0.8527987706390019

xiaoxij | 0.012629364336141216 | 0.8654281349751431

xiaoxiaohe | 0.012557926504372248 | 0.8779860614795154

parul | 0.01189359470607216 | 0.8898796561855876

tanster | 0.011891957052053951 | 0.9017716132376415

rkulkarni | 0.01154082756190496 | 0.9133124407995465

zhidwu | 0.011411921236515766 | 0.9247243620360623

mbarta | 0.010266373971291584 | 0.9349907360073539

pbalik | 0.00941122238976664 | 0.9444019583971206

petervarshavsky | 0.006921395503412272 | 0.9513233539005329

nash | 0.006823244473963957 | 0.9581465983744969

tdean | 0.006395596257061733 | 0.9645421946315587

kdooley | 0.005955542002462856 | 0.9704977366340215

codyg | 0.00556986191447758 | 0.9760675985484991

angelc | 0.005503993954602632 | 0.9815715925031018

melissaf | 0.00510485674852314 | 0.9866764492516249

tomb | 0.002650755558001851 | 0.9893272048096268

bfong | 0.0019693749176145645 | 0.9912965797272414

abraham | 0.0015981535389241062 | 0.9928947332661655

Ranked people list (top 30, for tristan) from v2 in the sliding window

ending at 2019-08-05 00:15:52.260147:

user email|weight|cweight

andrea | 0.1642984373492097 | 0.1642984373492097

micah | 0.10707656525805181 | 0.27137500260726155

clin | 0.07984586693184857 | 0.35122086953911014

pwilf | 0.05392623126541343 | 0.40514710080452354

amelia | 0.05310330432027114 | 0.4582504051247947

agong | 0.05276949537777544 | 0.5110199005025702

xiaoxiaohe | 0.04757350978741538 | 0.5585934102899855

clarence | 0.042281851763121846 | 0.6008752620531074

xugangye | 0.04121954078409379 | 0.6420948028372011

anaid | 0.03051075555573856 | 0.6726055583929397

saraht | 0.026957468759837514 | 0.6995630271527772

angelc | 0.022203056732966464 | 0.7217660838857436

petervarshavsky | 0.0181022616698739 | 0.7398683455556175

abraham | 0.016989385032663555 | 0.7568577305882811

melissaf | 0.015226121886291712 | 0.7720838524745728

chiyuanm | 0.014781864305940353 | 0.7868657167805131

jisha | 0.014122873364623317 | 0.8009885901451365

sunny | 0.013363522356168372 | 0.8143521125013048

ivaldez | 0.013178460068428521 | 0.8275305725697333

nbern | 0.012824380905852436 | 0.8403549534755858

dsaiteja | 0.011216400493252648 | 0.8515713539688384

alexk | 0.011202070846075059 | 0.8627734248149135

mtikir | 0.009841118029077204 | 0.8726145428439906

parul | 0.009758064789453905 | 0.8823726076334446

timo | 0.009190861921059345 | 0.8915634695545039

nicklarson | 0.007967186794994145 | 0.8995306563494981

edwardd | 0.00723031477785517 | 0.9067609711273533

agarg | 0.006761816590643066 | 0.9135227877179963

carola | 0.006680616818861988 | 0.9202034045368583

mhmehkri | 0.006381492968911355 | 0.9265848975057697

Results: snapshots of metrics (ndcgs)

ndcgs of past k = 5 days for User 1661098624 ([email protected])

endtime of sliding win | V1 | V2

2019-08-03 01:50:37.815809 | 0.735623740468 | 0.750020541642

2019-08-02 01:50:37.815809 | 0.735230662198 | 0.945109849755

2019-08-01 01:50:37.815809 | 0.733876443222 | 0.937978722678

2019-07-31 01:50:37.815809 | 0.724431950821 | 0.867913398768

2019-07-30 01:50:37.815809 | 0.764167423296 | 0.870846024408



2019-08-03 01:47:37.217575 | 0.786796580126 | 0.777014858654

2019-08-02 01:47:37.217575 | 0.905060925235 | 0.839147249726

2019-08-01 01:47:37.217575 | 0.911971768517 | 0.799122436903

2019-07-31 01:47:37.217575 | 0.878533639404 | 0.925333126232

2019-07-30 01:47:37.217575 | 0.777720060311 | 0.855721826805



2019-08-03 01:51:52.399001 | 0.694083367577 | 0.841302895745

2019-08-02 01:51:52.399001 | 0.808621039132 | 0.847709649087

2019-08-01 01:51:52.399001 | 0.628629499506 | 0.847917382845

2019-07-31 01:51:52.399001 | 0.830742404744 | 0.84576135313

2019-07-30 01:51:52.399001 | 0.816769894458 | 0.866249126224



2019-08-03 01:53:47.630918 | 0.681000888483 | 0.814429157133

2019-08-02 01:53:47.630918 | 0.729604276209 | 0.794962227697

2019-08-01 01:53:47.630918 | 0.757893150376 | 0.818128208669

2019-07-31 01:53:47.630918 | 0.754213894943 | 0.864687060097

2019-07-30 01:53:47.630918 | 0.634731186431 | 0.604825867604



2019-08-03 02:07:01.724797 | 0.772179425767 | 0.892205521755

2019-08-02 02:07:01.724797 | 0.870605330305 | 0.914610611072

2019-08-01 02:07:01.724797 | 0.882298583019 | 0.925157803287

2019-07-31 02:07:01.724797 | 0.839413691601 | 0.885134903683

2019-07-30 02:07:01.724797 | 0.71975885378 | 0.856360343088



2019-08-03 02:10:28.705517 | 0.732041452909 | 0.867086975214

2019-08-02 02:10:28.705517 | 0.686567320964 | 0.667970026294

2019-08-01 02:10:28.705517 | 0.684771777403 | 0.632646208909

2019-07-31 02:10:28.705517 | 0.763818457892 | 0.679240526089

2019-07-30 02:10:28.705517 | 0.843210314392 | 0.835118481106



2019-08-03 02:09:19.193196 | 0.686673617945 | 0.626478912524

2019-08-02 02:09:19.193196 | 0.688637136557 | 0.633737442376

2019-08-01 02:09:19.193196 | 0.537644923873 | 0.511530518122

2019-07-31 02:09:19.193196 | 0.541603314645 | 0.575550832774

2019-07-30 02:09:19.193196 | 0.569717576754 | 0.668444489698



2019-08-03 02:17:06.978550 | 0.554853316836 | 0.860480540981

2019-08-02 02:17:06.978550 | 0.581684101025 | 0.865655341359

2019-08-01 02:17:06.978550 | 0.614974881651 | 0.867432658312

2019-07-31 02:17:06.978550 | 0.727975955715 | 0.829696037161

2019-07-30 02:17:06.978550 | 0.76298137265 | 0.844222712538

Results: snapshots of metrics (ndcgs)ndcgs of past k = 5 days for User 1661098624 ([email protected])

endtime of sliding win | ndcg_v1, ndcg_v2 | ndcg@10_v1, ndcg@10_v2

2019-08-03 05:37:54.408302 | 0.741478605156, 0.750255809082 | 0.404484997402, 0.584174402365

2019-08-02 05:37:54.408302 | 0.735247422602, 0.945109849755 | 0.372049326559, 0.905332230591

2019-08-01 05:37:54.408302 | 0.733716180402, 0.937934848204 | 0.423697431955, 0.884340680304

2019-07-31 05:37:54.408302 | 0.724500589069, 0.868065094291 | 0.418620166195, 0.806773017956

2019-07-30 05:37:54.408302 | 0.761761921447, 0.934852969397 | 0.506800781263, 0.880765666153



2019-08-03 05:39:39.744563 | 0.786479025846, 0.776056830545 | 0.507071972701, 0.523722111178

2019-08-02 05:39:39.744563 | 0.907005030559, 0.761872753493 | 0.796572887509, 0.491070562808

2019-08-01 05:39:39.744563 | 0.910515315567, 0.79796567697 | 0.816509078754, 0.505527639921

2019-07-31 05:39:39.744563 | 0.883740259487, 0.934990071068 | 0.686515622787, 0.815287541277

2019-07-30 05:39:39.744563 | 0.790326494543, 0.873236125411 | 0.637905881537, 0.752585795856



2019-08-03 05:40:44.160206 | 0.682934864522, 0.849149958146 | 0.321866346413, 0.637701283243

2019-08-02 05:40:44.160206 | 0.810976833653, 0.847079797378 | 0.581423874269, 0.712620647346

2019-08-01 05:40:44.160206 | 0.6174988808, 0.847983706817 | 0.272378627677, 0.763401286257

2019-07-31 05:40:44.160206 | 0.849816807418, 0.832901048485 | 0.816787028762, 0.586128832723

2019-07-30 05:40:44.160206 | 0.805582402467, 0.860534151429 | 0.614456460994, 0.680905855325



2019-08-03 05:45:35.952885 | 0.663134480068, 0.778003795767 | 0.164566329575, 0.461508444581

2019-08-02 05:45:35.952885 | 0.716630052259, 0.736272024954 | 0.39844010026, 0.528866924036

2019-08-01 05:45:35.952885 | 0.751262983147, 0.817704243324 | 0.652248327521, 0.717524020813

2019-07-31 05:45:35.952885 | 0.739458148141, 0.696790816968 | 0.542890365053, 0.453308771948

2019-07-30 05:45:35.952885 | 0.721675735006, 0.813697630577 | 0.599124466787, 0.596546111



2019-08-03 05:50:14.847980 | 0.767331866579, 0.892800250145 | 0.497775920051, 0.694894061908

2019-08-02 05:50:14.847980 | 0.870536877274, 0.914589883545 | 0.656993614746, 0.761457527594

2019-08-01 05:50:14.847980 | 0.87985221576, 0.926886317483 | 0.733520885316, 0.840236268236

2019-07-31 05:50:14.847980 | 0.8395597799, 0.88542518574 | 0.535915338914, 0.651252514057

2019-07-30 05:50:14.847980 | 0.75019016323, 0.825044874263 | 0.403019005615, 0.627613593343



2019-08-03 05:52:48.772541 | 0.718690569082, 0.817372000832 | 0.411187025898, 0.610250616946

2019-08-02 05:52:48.772541 | 0.684255141249, 0.669350134307 | 0.284272326876, 0.321000992364

2019-08-01 05:52:48.772541 | 0.684848412615, 0.639837495721 | 0.386217117913, 0.265787556019

2019-07-31 05:52:48.772541 | 0.763818457892, 0.679240526089 | 0.629324604866, 0.32648269341

2019-07-30 05:52:48.772541 | 0.828300393827, 0.822082666866 | 0.678210712319, 0.715663613812



2019-08-03 05:54:08.929864 | 0.686673617945, 0.626478912524 | 0.296266051063, 0.36009610058

2019-08-02 05:54:08.929864 | 0.691272401501, 0.634105938828 | 0.293420682956, 0.387048554258

2019-08-01 05:54:08.929864 | 0.537444725469, 0.514107915384 | 0.17284272596, 0.257247136876

2019-07-31 05:54:08.929864 | 0.541603314645, 0.551630330236 | 0.207021281403, 0.390093557075

2019-07-30 05:54:08.929864 | 0.575512860044, 0.676457959251 | 0.193571079826, 0.416777154753



2019-08-03 05:57:31.896031 | 0.550765032505, 0.860524972014 | 0.0425125728639, 0.744570483031

2019-08-02 05:57:31.896031 | 0.580542020338, 0.865834785785 | 0.133969118214, 0.799170285955

2019-08-01 05:57:31.896031 | 0.615286705257, 0.867349399039 | 0.147931422575, 0.767387602874

2019-07-31 05:57:31.896031 | 0.727975955715, 0.829700461747 | 0.376567530875, 0.58677924639

2019-07-30 05:57:31.896031 | 0.734340668959, 0.831346842056 | 0.37130645551, 0.599726893728

Results: conditional ranking• People ranking result: • Per-person activity-ranking result:

userEmail | weight | cweight

[email protected] | 14.553575087779661 | 14.553575087779661

[email protected] | 7.174627850741284 | 21.728202938520944

[email protected] | 6.431021627850276 | 28.159224566371222

[email protected] | 6.240549105820271 | 34.39977367219149

[email protected] | 5.629892556007677 | 40.02966622819917

[email protected] | 5.152767744329558 | 45.18243397252873

[email protected] | 4.556403903918814 | 49.73883787644754

[email protected] | 4.322416200559127 | 54.061254077006666

[email protected] | 3.6721259196768523 | 57.73337999668352

[email protected] | 3.584687891084671 | 61.31806788776819

[email protected] | 2.6439897853056946 | 63.96205767307388

[email protected] | 2.409939344363988 | 66.37199701743786

[email protected] | 2.2962409319224757 | 68.66823794936033

[email protected] | 2.2096988981026557 | 70.87793684746299

[email protected] | 2.1710133935720974 | 73.04895024103509

[email protected] | 1.9906290564061258 | 75.03957929744121

[email protected] | 1.71048097326842 | 76.75006027070962

[email protected] | 1.2926829758104093 | 78.04274324652003

[email protected] | 1.0411423027244762 | 79.08388554924451

[email protected] | 0.935382018454251 | 80.01926756769876

[email protected] | 0.9146631168628628 | 80.93393068456163

[email protected] | 0.9027952912807606 | 81.8367259758424

[email protected] | 0.9009932818372495 | 82.73771925767964

[email protected] | 0.856353053582589 | 83.59407231126224

[email protected] | 0.7948458056335114 | 84.38891811689575

[email protected] | 0.7830159757122976 | 85.17193409260805

[email protected] | 0.7586413604126822 | 85.93057545302074

[email protected] | 0.7469200751231991 | 86.67749552814394

[email protected] | 0.7236363535475026 | 87.40113188169144

[email protected] | 0.6266019139527803 | 88.02773379564422

userEmail | personCWeight | docName | docCWeight | action | timestamp | eventSc

[email protected] | 14.553575087779661 | Assistant Experiment Tracker.gsheet | 6.643746690044808 | EDIT | 2019-05-28 21:22:36 | 21.005376198719006

[email protected] | 14.553575087779661 | � Content Suggestions Weekly Sync | 26.752844900577436 | PAPER_FOLLOW_EDIT | 2019-05-28 19:25:17 | 18.621607222464473

[email protected] | 14.553575087779661 | � Content Suggestions Weekly Sync | 26.752844900577436 | PAPER_FOLLOW_VIEW | 2019-05-28 19:06:23 | 18.618490347954765

[email protected] | 14.553575087779661 | Activity Highlights Weekly Meeting | 17.509939345042564 | PAPER_FOLLOW_VIEW | 2019-05-28 22:35:58 | 17.46495046261808

[email protected] | 14.553575087779661 | � Notes: Activity Highlights Input Metric | 48.743594120476615 | PAPER_FOLLOW_VIEW | 2019-05-28 19:23:23 | 17.350463256399824

[email protected] | 14.553575087779661 | Assistant A/B Experiment: File Metadata Features | 92.5995592782214 | PAPER_FOLLOW_EDIT | 2019-05-24 23:06:02 | 17.225458392090292

[email protected] | 14.553575087779661 | Assistant A/B Experiment: File Metadata Features | 92.5995592782214 | PAPER_FOLLOW_VIEW | 2019-05-24 21:32:59 | 17.214830636985972

[email protected] | 14.553575087779661 | Assistant A/B Experiment: File Metadata Features | 92.5995592782214 | PAPER_FOLLOW_VIEW | 2019-05-24 21:32:47 | 17.214807793879892

[email protected] | 14.553575087779661 | :spotlight: Spec: Activity Highlights | 88.26454877404272 | PAPER_COMMENT | 2019-05-24 04:17:51 | 13.502087505435794

[email protected] | 14.553575087779661 | :spotlight: Spec: Activity Highlights | 88.26454877404272 | PAPER_FOLLOW_VIEW | 2019-05-24 04:15:34 | 13.187007471937175

[email protected] | 14.553575087779661 | :spotlight: Spec: Activity Highlights | 88.26454877404272 | PAPER_FOLLOW_VIEW | 2019-05-24 04:14:48 | 13.186919906697197

[email protected] | 21.728202938520944 | Antenna support for Single-file edit ACLs and Righteous links | 30.42647427204275 | PAPER_COMMENT | 2019-05-31 17:33:38 | 20.88013977884638

[email protected] | 21.728202938520944 | Assistant Tray Dark Launch Debrief and G1.0 Launch Configuration | 36.56947864583676 | PAPER_FOLLOW_VIEW | 2019-05-31 18:40:25 | 17.951473767741625





[email protected] | 21.728202938520944 | � Content Suggestions Weekly Sync | 26.752844900577436 | PAPER_FOLLOW_VIEW | 2019-05-28 22:15:55 | 17.32391468189469

[email protected] | 21.728202938520944 | Spec: Folder Suggestions in Collab Browse Sidebar [Gemini 1.1] | 97.72418958954184 | PAPER_COMMENT | 2019-05-23 22:44:07 | 15.98950170492205

[email protected] | 21.728202938520944 | :spotlight: Spec: Activity Highlights | 88.26454877404272 | PAPER_COMMENT | 2019-05-23 17:35:41 | 15.707624026375555

[email protected] | 21.728202938520944 | Share events → Recents Investigation | 87.34629020588005 | PAPER_FOLLOW_VIEW | 2019-05-23 17:33:36 | 15.241806115550657

[email protected] | 28.159224566371222 | Share events investigation | 65.3398844928685 | PAPER_FOLLOW_VIEW | 2019-05-31 06:35:10 | 20.089939888877712




[email protected] | 28.159224566371222 | :link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | PAPER_FOLLOW_EDIT | 2019-05-29 21:03:59 | 19.122445182071527

[email protected] | 28.159224566371222 | :link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | PAPER_COMMENT | 2019-05-29 20:52:15 | 19.111181535709253



[email protected] | 28.159224566371222 | :link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | PAPER_FOLLOW_VIEW | 2019-05-29 18:55:41 | 18.91809781458939






[email protected] | 34.39977367219149 | :Gemini: Gemini 1.1 UX changes | 44.231189402360144 | PAPER_COMMENT | 2019-05-24 20:34:25 | 19.348433424759655

[email protected] | 34.39977367219149 | ✏ Content Spec: activity highlights | 63.542760990211974 | PAPER_FOLLOW_EDIT | 2019-05-31 20:07:11 | 18.394325149553634

[email protected] | 34.39977367219149 | ✏ Content Spec: activity highlights | 63.542760990211974 | PAPER_COMMENT | 2019-05-31 16:30:12 | 18.369542283047608

Results: conditional ranking

• Doc ranking result:

• Per-doc activity-ranking result:

docName | weight | cweight

Assistant Experiment Tracker.gsheet | 6.643746690044808 | 6.643746690044808

:link-winning: Spec: Shared Content in Feeds | 5.9836651420018185 | 12.627411832046626

Activity Highlights Weekly Meeting | 4.882527512995938 | 17.509939345042564

⭐ Collab Browse 6/11 Manual Opt-in GTM Details | 4.748542062274088 | 22.25848140731665

� Content Suggestions Weekly Sync | 4.494363493260785 | 26.752844900577436

Antenna support for Single-file edit ACLs and Righteous links | 3.673629371465315 | 30.42647427204275

Notifications | 3.0767345290671964 | 33.50320880110995

Assistant Tray Dark Launch Debrief and G1.0 Launch Configuration | 3.066269844726813 |

36.56947864583676

�♀ Sprint Calendar (drl/sprints) | 2.604754314413174 | 39.17423296024993

User Model Product Use Cases | 2.5970630042639296 | 41.77129596451386

:Gemini: Gemini 1.1 UX changes | 2.459893437846286 | 44.231189402360144

Assistant Desk Move | 2.2731700000209307 | 46.50435940238108

� Notes: Activity Highlights Input Metric | 2.239234718095542 | 48.743594120476615

Defining Frequent Collaborators | 1.9833543180857498 | 50.72694843856237

Assistant Tray Experiment Analysis Test #4 (V72) | 1.8945504065592016 | 52.62149884512157

⭐ Collab Browse/Assistant Tray M5 GTM Details | 1.8716020228374546 | 54.49310086795903

Metadata for Assistant Service | 1.8211093567272696 | 56.3142102246863

Gemini 1.0 informal retrospective input | 1.8170717259857303 | 58.131281950672026

Collab Plat Gemini 1.1 Requirements and Commitments | 1.8043666122008297 | 59.935648562872856

� Adam’s Office Hours Sign-up Sheet | 1.8038686380962674 | 61.739517200969125

✏ Content Spec: activity highlights | 1.8032437892428461 | 63.542760990211974

Share events investigation | 1.7971235026565349 | 65.3398844928685

Smarter Smart Sync: Success optimization | 1.7287637871184243 | 67.06864827998693

[SP4-5] Assistant Space / Tray Projects | 1.6376690423249551 | 68.70631732231189

SOA Extraction for Assistant | 1.5787157766002344 | 70.28503309891212

PRODUCT REVIEW 5.29 | 1.5290907037120531 | 71.81412380262418

� Notifications Types | 1.5256104844113967 | 73.33973428703558

Notifications at Gemini 1.1 | 1.3071430824928094 | 74.6468773695284

FAIL_TO_GET_PAPER_TITLE | 1.3066206507802487 | 75.95349802030864

desktop-web Gemini drafts | 1.299434853342988 | 77.25293287365163

docName | docCWeight | userEmail | personCWeight | action | timestamp | eventSc

Assistant Experiment Tracker.gsheet | 6.643746690044808 | [email protected] | 14.553575087779661 | EDIT | 2019-05-28 21:22:36 | 21.005376198719006

Antenna support for Single-file edit ACLs and Righteous links | 30.42647427204275 | [email protected] | 21.728202938520944 | PAPER_COMMENT | 2019-05-31 17:33:38 | 20.88013977884638

Assistant Tray Experiment Analysis Test #4 (V72) | 52.62149884512157 | [email protected] | 45.18243397252873 | PAPER_COMMENT | 2019-05-31 23:31:33 | 20.258820875728862

Share events investigation | 65.3398844928685 | [email protected] | 28.159224566371222 | PAPER_FOLLOW_VIEW | 2019-05-31 06:35:10 | 20.089939888877712



PRODUCT REVIEW 5.29 | 71.81412380262418 | [email protected] | 68.66823794936033 | PAPER_COMMENT | 2019-05-31 17:18:45 | 20.001933015515196


� Notes: Activity Highlights Input Metric | 48.743594120476615 | [email protected] | 57.73337999668352 | PAPER_FOLLOW_VIEW | 2019-05-28 18:41:07 | 19.730287373221678

desktop-web Gemini drafts | 77.25293287365163 | [email protected] | 49.73883787644754 | PAPER_COMMENT | 2019-05-29 16:25:20 | 19.504147469344762

Defining Frequent Collaborators | 50.72694843856237 | [email protected] | 73.04895024103509 | PAPER_FOLLOW_VIEW | 2019-05-29 19:46:12 | 19.407124028454803

Assistant Tray Dark Launch Debrief and G1.0 Launch Configuration | 36.56947864583676 | [email protected] | 70.87793684746299 | PAPER_FOLLOW_VIEW | 2019-05-30 16:57:05 | 19.3945047

Notifications at Gemini 1.1 | 74.6468773695284 | [email protected] | 63.96205767307388 | PAPER_FOLLOW_VIEW | 2019-05-28 21:59:16 | 19.383731939229413

:Gemini: Gemini 1.1 UX changes | 44.231189402360144 | [email protected] | 34.39977367219149 | PAPER_COMMENT | 2019-05-24 20:34:25 | 19.348433424759655

⭐ Collab Browse/Assistant Tray M5 GTM Details | 54.49310086795903 | [email protected] | 75.03957929744121 | PAPER_FOLLOW_VIEW | 2019-05-31 06:01:53 | 19.160806099845

⭐ Collab Browse/Assistant Tray M5 GTM Details | 54.49310086795903 | [email protected] | 75.03957929744121 | PAPER_FOLLOW_VIEW | 2019-05-31 04:27:35 | 19.15003557532767

:link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | [email protected] | 28.159224566371222 | PAPER_FOLLOW_EDIT | 2019-05-29 21:03:59 | 19.122445182071527

:link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | [email protected] | 28.159224566371222 | PAPER_COMMENT | 2019-05-29 20:52:15 | 19.111181535709253



Jeffrey’s Onboarding Doc | 90.25140921683858 | [email protected] | 81.8367259758424 | PAPER_FOLLOW_VIEW | 2019-05-30 21:34:25 | 18.933130897476154

:link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | [email protected] | 28.159224566371222 | PAPER_FOLLOW_VIEW | 2019-05-29 18:55:41 | 18.91809781458939

� Content Suggestions Weekly Sync | 26.752844900577436 | [email protected] | 61.31806788776819 | PAPER_FOLLOW_VIEW | 2019-06-02 17:07:52 | 18.813827218782745

� Notifications Types | 73.33973428703558 | [email protected] | 49.73883787644754 | PAPER_FOLLOW_EDIT | 2019-05-30 00:05:33 | 18.76794566409416

� Content Suggestions Weekly Sync | 26.752844900577436 | [email protected] | 14.553575087779661 | PAPER_FOLLOW_EDIT | 2019-05-28 19:25:17 | 18.621607222464473


Jeffrey’s Onboarding Doc | 90.25140921683858 | [email protected] | 81.8367259758424 | PAPER_FOLLOW_VIEW | 2019-05-28 21:53:29 | 18.606367879351993


Activity Highlights Weekly Meeting | 17.509939345042564 | [email protected] | 78.04274324652003 | PAPER_COMMENT | 2019-05-28 19:18:21 | 18.399371461406364

✏ Content Spec: activity highlights | 63.542760990211974 | [email protected] | 34.39977367219149 | PAPER_FOLLOW_EDIT | 2019-05-31 20:07:11 | 18.394325149553634

� Content Suggestions Weekly Sync | 26.752844900577436 | [email protected] | 66.37199701743786 | PAPER_FOLLOW_EDIT | 2019-05-28 22:01:28 | 18.392680617624272


✏ Content Spec: activity highlights | 63.542760990211974 | [email protected] | 54.061254077006666 | PAPER_COMMENT | 2019-05-31 19:10:17 | 18.387826285873505

✏ Content Spec: activity highlights | 63.542760990211974 | [email protected] | 54.061254077006666 | PAPER_FOLLOW_VIEW | 2019-05-31 19:08:37 | 18.387635926656163

:link-winning: Spec: Shared Content in Feeds | 12.627411832046626 | [email protected] | 40.02966622819917 | PAPER_FOLLOW_VIEW | 2019-05-24 18:21:02 | 18.378151751277812


✏ Content Spec: activity highlights | 63.542760990211974 | [email protected] | 34.39977367219149 | PAPER_COMMENT | 2019-05-31 16:30:12 | 18.369542283047608

Assistant Services eng workstreams | 86.42623040394479 | [email protected] | 76.75006027070962 | PAPER_FOLLOW_EDIT | 2019-05-23 22:05:21 | 18.299010003486075

Defining Frequent Collaborators | 50.72694843856237 | [email protected] | 83.59407231126224 | PAPER_FOLLOW_VIEW | 2019-05-28 18:42:02 | 18.246190764081767

Activity Highlights [Designs Explored] | 85.42143999176196 | [email protected] | 34.39977367219149 | PAPER_FOLLOW_VIEW | 2019-05-30 21:52:38 | 18.160470729803862

Activity Highlights [Designs Explored] | 85.42143999176196 | [email protected] | 54.061254077006666 | PAPER_COMMENT | 2019-05-30 18:06:48 | 18.13467705585374





Assistant Tray Dark Launch Debrief and G1.0 Launch Configuration | 36.56947864583676 | [email protected] | 21.728202938520944 | PAPER_FOLLOW_VIEW | 2019-05-31 18:40:25 | 17.95147

GPR3: Assistant Tray (Intelligence) | 89.65022844871393 | [email protected] | 75.03957929744121 | PAPER_FOLLOW_VIEW | 2019-05-31 00:40:42 | 17.92790480210614

To be continued …

• Metrics calculation on grouped results

• Prior statistics (i.e. �, / �� for � � , �|�, � , �|�, � , � � )

• User embedding

• � �

• � �

• l � , where l stands for keyword token

activity highlights dd - cis.jhu.edu · 0,stellax,paper_follow_edit,important_person,ml for...

Documents