INDIAN INSTITUTE OF TECHNOLOGY, BOMBAY

Ekalavya Summer Internship, 2018

Event Logging System for Collaborative

Communities

Divyansh Sharma Kartik Verma Rahul Raj Vishrut Jetly

NIT-Hamirpur NIT-Hamirpur NIT-Patna DA-IICT

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 1 / 28

Outline

1 Introduction

2 Motivation

3 Technologies Used

4 Work Flow

5 Capturing theEvents

6 Creating the Log

7 Storing Logs inElasticsearch

8 Event Log RESTAPI

9 User Analytics

10 Dockerization andSource Code

11 Selenium Testing

12 Problems Solved

13 Future Scope

14 References

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 2 / 28

Introduction

Introduction

What is Event Logging ?

Logging or storing the activities done by user of the system e.g.login, view a community, view article, edit article etc.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 3 / 28

Motivation

Motivation

The event logs will be used to create a view history for everyuser which will help in training the Recommendation System.

Providing analytics to the users like

Top communities and articlesTrending communities and articles

which will assist in users’decision making.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 4 / 28

Technologies Used

Technologies Used

Django Middleware

ELK

C3.js

Django REST framework

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 5 / 28

Technologies Used

Technologies Used

Django Middleware

Every activity that gen-erates a http request tothe server first passesthrough these middle-wares which attach andextract some informa-tion from the requeste.g. session id, authen-tication tokens etc.

C3.js

C3.js is used to im-plement high quality,interactive charts forvisualisation. Chartsbuilt using this librarycan be rendered in realtime.

Django RESTFramework

Django REST frame-work is a powerfuland flexible toolkit forbuilding Web APIs. Itsa way to expose partsof the application tothird-parties

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 6 / 28

Technologies Used

Technologies Used

ELK

ELK is acronym for three open source projects

Elasticsearch

Logstash

Kibana

Logstash [3] receives the data from the files or through the httprequests, applies filters, and then Elasticsearch [1] stores the logs,which can be later retrieved using Elasticsearch API.Kibana [2] is used for visualisations and analytics.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 7 / 28

Work Flow

Work Flow Diagram

Figure: Work Flow Diagram

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 8 / 28

Capturing the Events

Capturing the Events

Figure: Django Middleware

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 9 / 28

Capturing the Events

Capturing the Events

The request passes through all the middlewares and finallymoves through the eventlog middleware.

The eventlog middleware [4] captures all the information fromthe request.

Benefit of Middleware

Middleware keeps the Eventlog system as a completely separatemodule or app.

It can be reused in other django projects.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 10 / 28

Creating the Log

Creating the Log

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 11 / 28

Creating the Log

Important Points

The size of the bucket holding logs is not fixed.

The function which processes the dictionary present in bucketruns in a separate daemon thread.

Logs can be stored either in file or in Elasticsearch but here weare showing it only with Elasticsearch because storing logs in afile are not useful.

Logs are passed to ELK in JSON format.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 12 / 28

Creating the Log

Log Structure

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 13 / 28

Storing Logs in Elasticsearch

Storing Logs in Elasticsearch

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 14 / 28

Storing Logs in Elasticsearch

Important Points

Elasticsearch, Logstash and Kibana runs on separate ports insidea docker.

The name of the index which stores the logs is logs but can bechanged in settings.

Kibana is available only for the Super Admin users.

Elasticsearch uses DSL query syntax built over Apache luceneQuery Syntax parser for searching.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 15 / 28

Event Log REST API

Event Log REST API

Event log System provides a feature rich REST based API whichis a wrapper over the Elasticsearch search API.

This API is used by Recommendation System and for UserAnalytics

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 16 / 28

Event Log REST API

Features of REST API

The eventlog REST API is a feature rich API. The notable ones are:

Pagination

Sorting

Custom Fields

Filters based searching

Time Range Based searching

Aggregations or group by

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 17 / 28

User Analytics

User Analytics

User Analytics provides an overview of the activities of the user.

Some of the Analytics added are:

Trending articles in a community or group.

Most viewed articles in community.

User’s most viewed articles.

User’s own articles in various states.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 18 / 28

User Analytics

Some Snapshots

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 19 / 28

User Analytics

Some Snapshots

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 20 / 28

User Analytics

Some Snapshots

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 21 / 28

Dockerization and Source Code

Dockerization and Source Code

Components Dockerized:

Elasticsearch

Logstash

Kibana

Source Code Link:https://github.com/fresearchgroup/Collaboration-System/tree/eventlogs

ELK Docker Link:https://github.com/fresearchgroup/Collaboration-System-Event-Logs

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 22 / 28

Selenium Testing

Selenium Testing

Verified all events are logged.

Verified the log structure.

Tested the Event Log REST API.

Selenium Test Code Link:https://github.com/fresearchgroup/Collaboration-System-Selenium/tree/eventlogs

This Testing was done by the Notification System Team,Fundamental Research Group

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 23 / 28

Problems Solved

Problems Solved

Added the Event Logging System without affecting the existingsystem in any way using middleware.

Added a System which can be easily modified to add new eventsfor capture.

Provided a feature rich API for use by recommendation systemand for user analytics.

Added User Analytics to the current System.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 24 / 28

Future Scope

Future Scope

Adding browser events i.e. events that are generated in frontend, for instance

Window resize eventsScroll eventsThese events may provide some useful information about humanbehaviour

Adding Video related events

Adding more User Analytics

Add real time analytics whenever logs are updated

Storing logs in cloud storage.

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 25 / 28

References

References I

[1] elastic. Elasticsearch Reference and Docs. url:https://www.elastic.co/guide/en/elasticsearch/

reference/6.3/index.html (visited on 06/02/2018).

[2] elastic. Kibana Reference. url: https://www.elastic.co/guide/en/kibana/current/getting-

started.html (visited on 06/04/2018).

[3] elastic. Logstash Reference. url: https://www.elastic.co/guide/en/logstash/6.2/index.html

(visited on 06/03/2018).

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 26 / 28

References

References II

[4] Vitor Freitas. How to Create a Custom Django Middleware.url: https://simpleisbetterthancomplex.com/tutorial/2016/07/18/how-to-create-a-custom-django-

middleware.html (visited on 05/20/2018).

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 27 / 28

References

Thank You

Divyansh, Kartik, Rahul, Vishrut Event Logging System July 5, 2018 28 / 28