shipping & visualize your data with elk
TRANSCRIPT
![Page 1: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/1.jpg)
Shipping & Visualize Your Data With ELK
Adam Chen
![Page 2: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/2.jpg)
HELLO Everyone!I am AdamI am just an engineer like to share some experience with others.Thanks Gentoo let me familiar with Linux.
You can find me at @adaam
![Page 3: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/3.jpg)
Let’s Start This Session
![Page 4: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/4.jpg)
ELK? What !?
![Page 5: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/5.jpg)
Develop by
![Page 6: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/6.jpg)
Elastic Family
![Page 7: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/7.jpg)
ELK ?DevOps ?
![Page 8: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/8.jpg)
Log Still Play An Important Role
![Page 9: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/9.jpg)
Today’s characters
Logstash/Beat seriesShipping all of your log to where it should go, like ES, AWS, or just text.
ElasticsearchThe main part to store your data with High Availability.
KibanaVisualize will power your data. To know more about its value.
![Page 10: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/10.jpg)
Traditional Way to Collect Log
When error happened, administrator or RD/QA will need to login or write/use tool to grab log from each machines then analysis.
![Page 11: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/11.jpg)
Hey Bob, Please collect the error log to analyze.
OK, Boss.
BOSS
![Page 12: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/12.jpg)
![Page 13: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/13.jpg)
Hey Bob, Please collect the error log to analyze.
Traditional Way to Collect Log
![Page 14: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/14.jpg)
![Page 15: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/15.jpg)
Old Way to Collect Log
![Page 16: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/16.jpg)
How ELK Help ?
![Page 17: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/17.jpg)
Centralize Log To One Place
![Page 18: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/18.jpg)
Collect Log using ELK
![Page 19: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/19.jpg)
Introduce The E, The L and The K
![Page 20: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/20.jpg)
Logstash
An agent install at where log need to be collect.
Have much filter to process your log.Also Input/Output module.
Logstash Module
![Page 21: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/21.jpg)
Logstash Filters
Date, geoip, json, kv ...etc
GROK !!
Grok online tester
![Page 22: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/22.jpg)
Logstash DEMO
Input apache/nginx log then output to stdout, using local logstash.
![Page 23: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/23.jpg)
Elasticsearch
SImply a data store with near real time search
Store data in index, can by hours, day, week, month. Setting at Logstash.
Machine spec depends on data size.
![Page 24: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/24.jpg)
Elasticsearch Modules
They are improve ES function if you need.
Watcher, Shield, Marvel, Cloud-AWS
![Page 25: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/25.jpg)
Elasticsearch DEMO
API Webpage
![Page 26: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/26.jpg)
Kibana
Show your data to you understand.But you need to know your data fields first.
![Page 27: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/27.jpg)
Kibana Demo
Create some template to show (data pregen at ES? )
Real time insert data to ES and see from template pregen. (apache/nginx?)
![Page 28: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/28.jpg)
What else ?
![Page 29: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/29.jpg)
Push Metrics to elsewhere
![Page 30: Shipping & Visualize Your Data With ELK](https://reader034.vdocuments.net/reader034/viewer/2022042611/587196c11a28ab044e8b4d43/html5/thumbnails/30.jpg)
THANKS!Any questions?You can find me at:@adaam