1010-managing black friday logs bigdatadays · data your{beat} elasticsearch master nodes (3)...
TRANSCRIPT
![Page 1: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/1.jpg)
David Pilato Developer | Evangelist, @dadoonet
Managing your Black Friday Logs
![Page 2: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/2.jpg)
Data Platform Architectures
![Page 3: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/3.jpg)
@dadoonet sli.do/elastic3
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
![Page 4: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/4.jpg)
@dadoonet sli.do/elastic4
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
![Page 5: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/5.jpg)
@dadoonet sli.do/elastic5
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
Kibana
Instances (X)
![Page 6: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/6.jpg)
@dadoonet sli.do/elastic6
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
Logstash
Nodes (X)
Kibana
Instances (X)
![Page 7: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/7.jpg)
@dadoonet sli.do/elastic7
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Data Store
Web APIs
Social Sensors
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
Logstash
Nodes (X)
Kibana
Instances (X)
![Page 8: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/8.jpg)
@dadoonet sli.do/elastic8
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Data Store
Web APIs
Social Sensors
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
Logstash
Nodes (X)
Kibana
Instances (X)
NotificationQueues Storage Metrics
![Page 9: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/9.jpg)
@dadoonet sli.do/elastic9
The Elastic Journey of Data
Beats
Log Files
Metrics
Wire Data
your{beat}
Data Store
Web APIs
Social Sensors
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
Logstash
Nodes (X)Kafka
Redis
Messaging Queue
Kibana
Instances (X)
NotificationQueues Storage Metrics
![Page 10: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/10.jpg)
SaaS
Elastic Cloud
Metrics
Logging
APM
Site Search
App Search
Business Analytics
Elastic Stack
Kibana
Elasticsearch
Beats Logstash
Self Managed
Elastic CloudEnterprise
Standalone
Visualize & Manage
Store, Search, & Analyze
Ingest
EnterpriseSearch
SecurityAnalytics
Future
Deployment
Solutions
![Page 11: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/11.jpg)
ElasticsearchCluster Sizing
![Page 12: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/12.jpg)
@dadoonet sli.do/elastic12
Terminology
Cluster my_clusterServer 1
Node Ad1
d2d3
d4
d5
d6
d7
d8d9
d10
d11
d12
Index twitter
d6d3
d2
d5
d1
d4
Index logs
![Page 13: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/13.jpg)
@dadoonet sli.do/elastic13
Partition
Cluster my_clusterServer 1
Node Ad1
d2d3
d4
d5
d6
d7
d8d9
d10
d11
d12
Index twitter
d6d3
d2
d5
d1
d4
Index logs
Shards0
1
4
2
3
0
1
![Page 14: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/14.jpg)
@dadoonet sli.do/elastic14
Distribution
Cluster my_clusterServer 1
Node AServer 2
Node Btwitter shard P4
d1
d2d6
d5
d10
d12
twitter shard P2
twitter shard P1
logs shard P0
d2
d5d4
logs shard P1
d3
d4
d9
d7
d8
d11
twitter shard P3
twitter shard P0
d6d3
d1
![Page 15: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/15.jpg)
@dadoonet sli.do/elastic15
Replication
Cluster my_clusterServer 1
Node AServer 2
Node Btwitter shard P4
d1
d2d6
d5
d10
d12
twitter shard P2
twitter shard P1
logs shard P0
d2
d5d4
logs shard P1
d3
d4
d9
d7
d8
d11
twitter shard P3
twitter shard P0
twitter shard R4
d1
d2d6
d12
twitter shard R2
d5
d10
twitter shard R1
d6d3
d1
d6d3
d1
logs shard R0
d2
d5d4
logs shard R1
d3
d4
d9
d7
d8
d11
twitter shard R3
twitter shard R0
• Primaries • Replicas
![Page 16: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/16.jpg)
@dadoonet sli.do/elastic16
Replication
Cluster my_clusterServer 1
Node AServer 2
Node Btwitter shard P4
d1
d2d6
d12
twitter shard P2
d5
d10
twitter shard P1
d3
d4
d9twitter shard P3
d7
d8
d11
twitter shard P0
twitter shard R4
d1
d2d6
d12
twitter shard R2
d5
d10
twitter shard R1
logs shard P0
d6d3
d1
d6d3
d1
logs shard R0
d2
d5d4
logs shard R1
d3
d4
d9twitter shard R3
d7
d8
d11
twitter shard R0
• Primaries • Replicas
Server 3Node C
d2
d5d4
logs shard P1
![Page 17: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/17.jpg)
@dadoonet sli.do/elastic17
Scaling
Data
![Page 18: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/18.jpg)
@dadoonet sli.do/elastic18
Scaling
Data
![Page 19: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/19.jpg)
@dadoonet sli.do/elastic19
Scaling
Data
![Page 20: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/20.jpg)
@dadoonet sli.do/elastic20
Scaling
Big Data
... ...
![Page 21: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/21.jpg)
@dadoonet sli.do/elastic21
Scaling
• In Elasticsearch, shards are the working unit
• More data -> More shards
Big Data
... ...
But how many shards?
![Page 22: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/22.jpg)
@dadoonet sli.do/elastic22
How much data?
• ~1000 events per second
• 60s * 60m * 24h * 1000 events => ~87M events per day
• 1kb per event => ~82GB per day
• 3 months => ~7TB
![Page 23: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/23.jpg)
@dadoonet sli.do/elastic23
Shard Size
• It depends on many different factors
‒ document size, mapping, use case, kinds of queries being executed, desired response time, peak indexing rate, budget, ...
• After the shard sizing*, each shard should handle 45GB • Up to 10 shards per machine
* https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing
![Page 24: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/24.jpg)
@dadoonet sli.do/elastic24
How many shards?
• Data size: ~7TB
• Shard Size: ~45GB*
• Total Shards: ~160
• Shards per machine: 10*
• Total Servers: 16
* https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing
Cluster my_cluster
3 months of logs
...
![Page 25: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/25.jpg)
@dadoonet sli.do/elastic25
But...
• How many indices?
• What do you do if the daily data grows?
• What do you do if you want to delete old data?
![Page 26: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/26.jpg)
@dadoonet sli.do/elastic26
Time-Based Data
• Logs, social media streams, time-based events
• Timestamp + Data
• Do not change
• Typically search for recent events
• Older documents become less important
• Hard to predict the data size
![Page 27: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/27.jpg)
@dadoonet sli.do/elastic27
Time-Based Data
• Time-based Indices is the best option
‒ create a new index each day, week, month, year, ...
‒ search the indices you need in the same request
![Page 28: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/28.jpg)
@dadoonet sli.do/elastic28
Daily IndicesCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-20
![Page 29: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/29.jpg)
@dadoonet sli.do/elastic29
Daily IndicesCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-21
d6d3
d2
d5
d1
d4
logs-2019-02-20
![Page 30: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/30.jpg)
@dadoonet sli.do/elastic30
Daily IndicesCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-20
d6d3
d2
d5
d1
d4
logs-2019-02-22
d6d3
d2
d5
d1
d4
logs-2019-02-21
![Page 31: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/31.jpg)
@dadoonet 31
Templates
• Every new created index starting with 'logs-' will have
‒ 2 shards
‒ 1 replica (for each primary shard)
‒ 60 seconds refresh interval
PUT _template/logs { "template": "logs-*", "settings": { "number_of_shards": 2, "number_of_replicas": 1, "refresh_interval": "60s" } }
More on that later
![Page 32: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/32.jpg)
@dadoonet sli.do/elastic32
AliasCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-20
users
Application
logs-write
logs-read
![Page 33: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/33.jpg)
@dadoonet sli.do/elastic33
AliasCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-20
users
Application
logs-write
logs-readd6d3
d2
d5
d1
d4
logs-2019-02-21
![Page 34: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/34.jpg)
@dadoonet sli.do/elastic34
AliasCluster my_cluster
d6d3
d2
d5
d1
d4
logs-2019-02-20
users
Application
logs-write
logs-readd6d3
d2
d5
d1
d4
logs-2019-02-21d6d3
d2
d5
d1
d4
logs-2019-02-22
![Page 35: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/35.jpg)
Detour: Rollover API
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-rollover-index.html
![Page 36: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/36.jpg)
@dadoonet sli.do/elastic36
Do not Overshard
• 3 different logs
• 1 index per day each
• 1GB each
• 5 shards (default): so 200mb / shard vs 45gb
• 6 months retention
• ~900 shards for ~180GB • we needed ~4 shards!
don't keep default values! Cluster my_cluster
access-...
d6d3
d2
d5
d1
d4
application-...
d6d5
d9
d5
d1
d7
mysql-...
d10d59
d3
d5
d0
d4
![Page 37: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/37.jpg)
@dadoonet sli.do/elastic37
Scaling
Big Data
... ...1M users
But what happens if we have 2M users?
![Page 38: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/38.jpg)
@dadoonet sli.do/elastic38
Scaling
Big Data
... ...1M users
... ...1M users
![Page 39: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/39.jpg)
@dadoonet sli.do/elastic39
Scaling
Big Data
... ...1M users
... ...1M users
... ...1M users
![Page 40: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/40.jpg)
@dadoonet sli.do/elastic40
Scaling
Big Data
... ...
... ...
... ...
Users
![Page 41: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/41.jpg)
@dadoonet sli.do/elastic41
Shards are the working unit
• Primaries
‒ More data -> More shards
‒ write throughput (More writes -> More primary shards)
• Replicas
‒ high availability (1 replica is the default)
‒ read throughput (More reads -> More replicas)
![Page 42: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/42.jpg)
Detour: Shrink API
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-shrink-index.html
![Page 43: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/43.jpg)
Detour: Split API
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-split-index.html
![Page 44: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/44.jpg)
Optimal Bulk Size
![Page 45: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/45.jpg)
@dadoonet sli.do/elastic45
What is Bulk?
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
_____________________________________________
1000 log events
Beats
Logstash
Application
1000 index requests with 1 document
1 bulk request with 1000 documents
![Page 46: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/46.jpg)
@dadoonet sli.do/elastic46
What is the optimal bulk size?
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
_____________________________________________
1000 log events
Beats
Logstash
Application
4 * 250?
1 * 1000?
2 * 500?
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
![Page 47: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/47.jpg)
@dadoonet sli.do/elastic47
It depends...
• on your application (language, libraries, ...)
• document size (100b, 1kb, 100kb, 1mb, ...)
• number of nodes
• node size
• number of shards
• shards distribution
![Page 48: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/48.jpg)
@dadoonet sli.do/elastic48
Test it ;)
Elasticsearch
Master Nodes (3)
Ingest Nodes (X)
Data Nodes Hot (X)
Data Notes Warm (X)
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
_____________________________________________
1000000log events
Beats
Logstash
Application
4000 * 250-> 160s
1000 * 1000-> 155s
2000 * 500-> 164s
![Page 49: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/49.jpg)
@dadoonet sli.do/elastic49
Test it ;)
DATE=`date +%Y.%m.%d` LOG=logs/logs.txt
exec_test () { curl -s -XDELETE "http://USER:PASS@HOST:9200/logstash-$DATE" sleep 10 export SIZE=$1 time cat $LOG | ./bin/logstash -f logstash.conf }
for SIZE in 100 500 1000 3000 5000 10000; do for i in {1..20}; do exec_test $SIZE done; done;
input { stdin{} }
filter {}
output { elasticsearch { hosts => ["10.12.145.189"] flush_size => "${SIZE}" } }
In Beats set "bulk_max_size" in the output.elasticsearch
![Page 50: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/50.jpg)
@dadoonet 50
Test it ;)
• 2 node cluster (m3.large)
‒ 2 vCPU, 7.5GB Memory, 1x32GB SSD
• 1 index server (m3.large)
‒ logstash
‒ kibana# docs 100 500 1000 3000 5000 10000
time(s) 191.7 161.9 163.5 160.7 160.7 161.5
![Page 51: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/51.jpg)
Distribute the Load
![Page 52: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/52.jpg)
@dadoonet sli.do/elastic52
Avoid Bottlenecks
Elasticsearch
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
1000000log events
Beats
Logstash
Application
single nodeNode 1
Node 2round robin
output { elasticsearch { hosts => ["node1","node2"] } }
![Page 53: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/53.jpg)
@dadoonet sli.do/elastic53
Load Balancer
Elasticsearch
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
1000000log events
Beats
Logstash
Application
LB
Node 2
Node 1
![Page 54: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/54.jpg)
@dadoonet sli.do/elastic54
Coordinating-only Node
Elasticsearch
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
1000000log events
Beats
Logstash
Application
Node 3co-node
Node 2
Node 1
![Page 55: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/55.jpg)
@dadoonet sli.do/elastic55
Test it ;)
#docs time(s) 1000 5000 10000
NO Round Robin 163.5 160.7 161.5Round Robin 161.3 158.2 159.4
• 2 node cluster (m3.large)
‒ 2 vCPU, 7.5GB Memory, 1x32GB SSD
• 1 index server (m3.large)
‒ logstash (round robin configured) ‒ hosts => ["10.12.145.189", "10.121.140.167"] ‒ kibana
![Page 56: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/56.jpg)
Optimizing Disk IO
![Page 57: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/57.jpg)
@dadoonet 57
Durability
index a doc
time
lucene flush
buffer
index a docbuffer
index a docbuffer
buffer
segment
![Page 58: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/58.jpg)
@dadoonet 58
refresh_interval
• Dynamic per-index setting
• Increase to get better write throughput to an index
• New documents will take more time to be available for Search.
PUT logstash-2017.05.16/_settings { "refresh_interval": "60s" }
#docs time(s) 1000 5000 10000
1s refresh 161.3 158.2 159.460s refresh 156.7 152.1 152.6
![Page 59: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/59.jpg)
@dadoonet 59
Durability
index a doc
time
lucene flush
buffer
segment
trans_log
buffer
trans_log
buffer
trans_log
elasticsearch flush
doc
op
lucene commit segmentsegment
![Page 60: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/60.jpg)
@dadoonet 60
Translog fsync every 5s (1.7)
index a doc
buffer
trans_log
doc
op
index a doc
buffer
trans_log
doc
op
Primary
Replica
redundancy doesn’t help if all nodes lose power
![Page 61: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/61.jpg)
@dadoonet sli.do/elastic61
Translog fsync on every request
• For low volume indexing, fsync matters less
• For high volume indexing, we can amortize the costs and fsync on every bulk
• Concurrent requests can share an fsync
bulk 1
bulk 2
single fsync
![Page 62: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/62.jpg)
@dadoonet 62
Async Transaction Log
• index.translog.durability
‒ request (default)
‒ async
• index.translog.sync_interval (only if async is set)
• Dynamic per-index settings
• Be careful, you are relaxing the safety guarantees
#docs time(s) 1000 5000 10000
Request fsync 161.3 158.2 159.45s sync 152.4 149.1 150.3
![Page 63: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/63.jpg)
Final Remarks
![Page 64: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/64.jpg)
SaaS
Elastic Cloud
Metrics
Logging
APM
Site Search
App Search
Business Analytics
Elastic Stack
Kibana
Elasticsearch
Beats Logstash
Self Managed
Elastic CloudEnterprise
Standalone
Visualize & Manage
Store, Search, & Analyze
Ingest
EnterpriseSearch
SecurityAnalytics
Future
Deployment
Solutions
![Page 65: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/65.jpg)
@dadoonet sli.do/elastic65
Final Remarks• Primaries
‒ More data -> More shards
‒ Do not overshard!
• Replicas
‒ high availability (1 replica is the default)
‒ read throughput (More reads -> More replicas)
Big Data
... ...
... ...
... ...
Users
![Page 66: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/66.jpg)
@dadoonet 66
Final Remarks
• Bulk and Test
• Distribute the Load
• Refresh Interval
• Async Trans Log (careful)
#docs per bulk 1000 5000 10000
Default 163.5 160.7 161.5
RR+60s+Async5s 152.4 149.1 150.3
![Page 67: 1010-Managing Black Friday Logs BigDataDays · Data your{beat} Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Logstash Nodes (X) Kibana Instances](https://reader036.vdocuments.net/reader036/viewer/2022070909/5f910572a7566f78251c4977/html5/thumbnails/67.jpg)
David Pilato Developer | Evangelist, @dadoonet
Managing your Black Friday Logs