50 shades of elasticsearch · 50 shades of elasticsearch denis Ćutić @ infobip. use case. event...

67

50 shades of ElasticSearch Denis Ćutić @ Infobip

Upload: others

Post on 08-Jun-2020

3 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

50 shades of ElasticSearch

Denis Ćutić @ Infobip

Page 2: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

USE CASE

Page 3: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Event logs

Page 4: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Distributed processing

Page 5: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Multiple events per message

Page 6: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Additional information

Page 7: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Availability?

Page 8: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Retention

Page 9: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Options

Page 10: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Decision

Page 11: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

PoC

Page 12: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Hardware rental

Page 13: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

CLUSTER #1

Page 14: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

16 data nodes3 master nodes

40 TB

Page 15: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

30 days retention

Page 16: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Only message logs

Page 17: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Considerations before starting it up

Page 18: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Network connectivity / throughput

Page 19: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Plan capacity carefully

Page 20: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

RAID 0

Page 21: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Mount points

Page 22: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Configuration location

Page 23: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

ES version

Page 24: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Java version

Page 25: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Shard allocation

Page 26: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Once it’s started...

Page 27: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Logging

Page 28: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Kopf / Cerebro

Page 29: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Host / ES metrics exporter

Page 30: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Prometheus + Grafana

Page 31: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Slow logs & circuit breakers

Page 32: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Disable automatic index creation

Page 33: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Index / mapping templates

Page 34: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Changes take time to “propagate”

Page 35: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Remove _all field, disable field_data

Page 36: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Use aliases

Page 37: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Avoid custom routing

Page 38: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Security / access logs

Page 39: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Proxy

Page 40: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Throttling

Page 41: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Start filling the data...

Page 42: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Start dealing with challenges...

Page 43: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Rolling restart

Page 44: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Stop indexingDisable shard allocation

Flush

Page 45: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Recovery not always perfect

Page 46: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

“Manually” reallocate shards

Page 47: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Nodes falling down

Page 48: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Recovery slowing down whole cluster

Page 49: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Recovery speedup

Page 50: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Tune up the recovery settings

Page 51: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Stop indexing

Page 52: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Disable replicas

Page 53: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Allocation / task API

Page 54: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

CLUSTER #2

Page 55: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Message logsBilling data

Page 56: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

39 data nodes3 master nodes

50 TB

Page 57: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Traffic and archive data

Page 58: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Joining Kafka streams

Page 59: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Several approaches tried

Page 60: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Painless scripts

Page 61: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

GC issues

Page 62: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Result

Page 63: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Indexing: ~50k/s

Page 64: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Searches: ~1k/s

Page 65: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Recovery impact

Page 66: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Still tuning for optimal performance...

Page 67: 50 shades of ElasticSearch · 50 shades of ElasticSearch Denis Ćutić @ Infobip. USE CASE. Event logs. Distributed processing. Multiple events per message. Additional information

Thank you!

Infobip Products

Neo4j Integration with ElasticSearch - ElasticSearch Meetup

Infobip SMS Products

Elasticsearch - Introduction

Nexmo, Twilio, Plivo,Infobip | Company Showdown

Infobip Viber Connector For Salesforce Marketing …...Prerequisites To begin using Infobip Viber capabilities in Salesforce Marketing Cloud, you’ll need to: 1. Create Infobip account

HTTP Application Programming Interface SMPP ... - Infobip · 6/11/2014 · Infobip HTTP API can be used for sending SMS messages, collecting delivery reports, making Network Query

Infobip HTTP API and SMPP Specification

Dell EMC Search · above Elasticsearch. Elasticsearch cluster ports NGINX TCP/ HTTPS 9300– 9400 Ports for communicating with Elasticsearch (Index data nodes). Elasticsearch cluster

Личный кабинет Infobip

ElasticSearch Introduction

Business intelligence for the global carrier industry A ... · Infobip enables operators to optimise their network and revenue. Infobip sGate SMS firewall and monetization services

Elasticsearch: Accelerating the Django Admin · Elasticsearch Service elastic cloud Elasticsearch Reference + + + + + + + Elasticsearch Reference: 6.4 (current) Getting Started Set

Infobip - O que nos faz diferentes

Push Notification - Infobip

Elasticsearch Aggregations

Infobip Veselin Vuković Direktor prodaje

Elasticsearch Basics

Centili: Infobip mobile payments platform

Infobip - How Are We Different

Elasticsearch Terminology

Organizacijska klima u poduzeću Infobip d.o.o

Elasticsearch @ Keboola

Elasticsearch 5.0

Infobip SMS Connector For Salesforce Marketing Cloud · 2019-09-11 · Prerequisites To begin using Infobip SMS capabilities in Salesforce Marketing Cloud, you’ll need to: 1. Create

Elasticsearch 5 in Amazon Elasticsearch Service

ELASTICSEARCH INTRODUCTION › content › download › 6739 › 115193 › file...ELASTICSEARCH INTRODUCTION Zagreb, 27.03.2015. Kristijan Duvnjak & Mladen Maravi ć Elasticsearch

Elasticsearch & Docker

INFOBIP VIRTUAL CLASSROOM: Boost your Bip potential · Big picture of Infobip #generalIndustryTopic #A2P 2 PM 120’ Short introduction to Infobip world covering topics such as brief

Gamelab presentation - Infobip mobile payments

Elasticsearch Documentation

Elasticsearch Workshop

Elasticsearch 20150107

SOTI, MobileIron, AgreeYa Mobility, Infobip | Company Showdown

Elasticsearch Documentation€¦ · Elasticsearch Documentation, Release 1.4.0 Ofﬁcial low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related