Download - Introduction to ELK
![Page 1: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/1.jpg)
Introduction to ELK
20160316Yuhsuan_Chen
![Page 2: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/2.jpg)
2
ELK
• What is ELK?– Elasticsearch
It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents
– Logstash It is a tool to collect, process, and forward events and log messages
– KibanaIt provides visualization capabilities on top of the content indexed on an Elasticsearch cluster.
Server Filebeat
Server Filebeat
Logstash Elasticsearch Kibana Nginx User
![Page 3: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/3.jpg)
3
ELK
• Who use it?– SOUNDCLOUD
• https://www.elastic.co/assets/blt7955f90878661eec/case-study-soundcloud.pdf
– TANGO• https://www.elastic.co/assets/blt0dc7d9c62f60d38f/case-study-tango.pdf
– GOGOLOOK• http://www.slideshare.net/tw_dsconf/elasticsearch-kibana
– KKBOX• http://www.slideshare.net/tw_dsconf/kkbox-51962632
![Page 4: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/4.jpg)
4
Beats
• The Beats are open source data shippers that you install as agents on your servers to send different types of operational data to Elasticsearch.
• Beats can send data directly to Elasticsearch or send it to Elasticsearch via Logstash, which you can use to enrich or archive the data
![Page 5: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/5.jpg)
5
Logstash
• The ingestion workhorse for Elasticsearch and more– Horizontally scalable data processing pipeline with strong Elasticsearch and
Kibana synergy
• Pluggable pipeline architecture– Mix, match, and orchestrate different inputs, filters, and outputs to play in
pipeline harmony
• Community-extensible and developer-friendly plugin ecosystem– Over 200 plugins available, plus the flexibility of creating and contributing your
own
![Page 6: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/6.jpg)
6
Logstash
• Logstash Download Link– Linux
https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.2.2-1_all.deb
– Windowshttps://download.elastic.co/logstash/logstash/logstash-2.2.2.zip
• How to set up Logstash– http://howtorapeurjob.tumblr.com/post/140724250861/
![Page 7: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/7.jpg)
7
Logstash
• Start up– Linux:
service logstash restart
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/ -v
– Windows:{file_path}\bin\logstash.bat
• Parameter
-h or --help 相關資訊-t or --configtest 測試 config 是否正常-f 指定 config 檔案-v or --debug 輸出 debug 資訊-w 是以多少 thread 處理資料,預設 4-l 指定 log 輸出路徑
![Page 8: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/8.jpg)
8
Logstash - Configure
• Configure– 主結構分三個 input / filter / output ,同時都支援多種輸入輸出與外掛– Configure 可以放在同一份檔案也可以放在不同檔案, Logstash 會自動尋找相對應的資料
• Configure Structure
input { ….}
filter { ….}
output { ….}
1234567891011
![Page 9: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/9.jpg)
9
• stdin
• beats
Logstash - INPUT
Input { # 將 console 上的資料輸入 stdin{}}
12345
input { # 資料來源為 Beats , port 是 5044 ,定義的資料類型為 HTCLog beats{ port => 5044 type => "HTCLog" }
123456
![Page 10: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/10.jpg)
10
Logstash - INPUT
• generatorinput{ generator{ # 資料來源是以行方式呈現,如下面兩行 lines =>[ "02-23 10:19:58.588 1696 1750 D 1", "02-23 10:19:58.695 904 904 D 2" ] # 每一行的次數,這邊設定只輸入一次 count => 1 type => "device_log" }}
123456789101112
![Page 11: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/11.jpg)
11
Logstash - INPUT
• fileinput{ file{ # 指定輸入的檔案路徑,可用 array 方式記錄多個路徑 path => [ "/home/elasticadmin/Desktop/TestLog/*", "/home/elasticadmin/Desktop/TestLog/**/*.txt", ] # 用來監聽指定的檔案路徑下變更的時間,預設是 15 秒 discover_interval => 1 # 用來決定找檔案內容的方式,預設是默認結束的位置,如果要從頭開始找資訊就要設定為beginning start_position => "beginning"
# 用來記錄那些檔案是找過的,如果把檔案砍掉就可以讓 Logstash 重新查找之前的資料, Linux 如果要每次啟動都重新查找,要設定為 /dev/null sincedb_path => "/home/elasticadmin/Desktop/.sincedb_TestLog" }}
12345678910111213141516171819
![Page 12: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/12.jpg)
12
Logstash - FILTER
• grok
• Grok test webhttp://grokdebug.herokuapp.com/
filter { # 假設進來的資料 type 是 HTCLog 就已 grok 這個外掛來進行資料的篩選 if [type] == "HTCLog" { grok { match => {"message" => "(?<log_time>(0[0-9]+|1[0-2])-[0-6][0-9] %{TIME})\s+(?<pid>([0-9]+))\s+(?<ppid>([0-9]+))\s+(?<log_level>([Ww]))\s+(?<log>(.*)) "}
# 輸出時增加一個資料欄位叫 file_type ,內容為 kernel_log add_field => { "file_type" => "kernel_log" } } }}
123456789101112131415
![Page 13: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/13.jpg)
13
Logstash - FILTER
• mutate
filter { mutate { # 新增一個 field add_field => {"add word for host" => "Hello world, from %{host}" } # 新增一個 tag add_tag => {"add word for host" => "Hello world, from %{host}"} # 移除 host 這個 field remove_field => ["host"] # 移除 tag 這個 host remove_tag => ["host"] # 重新命名 host 這個 field rename => {"host" => "hostname"} # 重新將 message 的內容取代 replace => {"message" =>"The message was removed"} }}
12345678910111213141516
![Page 14: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/14.jpg)
14
Logstash - OUTPUT
• stdout
• file
output { # 將結果輸出到 console stdout {}}
1234
output{ file{ # 檔案輸出路徑 path =>"C:\Users\Yuhsuan_chen\Desktop\Elastic\logstash\result.gzip" # 將檔案輸出後,壓縮 gzip => true }}
123456789
![Page 15: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/15.jpg)
15
Logstash - OUTPUT
• elasticsearch
output { elasticsearch { # Elasticsearch 的 web link hosts => ["10.8.49.16:9200"] # 要將資料塞進的 index 名稱 index =>"%{[@metadata][beat]}-%{+YYYY.MM.dd}“ # 要將資料塞進的 type 名稱 document_type => "%{[@metadata][type]}" }}
12345678910
![Page 16: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/16.jpg)
16
Elasticsearch
Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time.
![Page 17: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/17.jpg)
17
Elasticsearch
• Elasticsearch on Ubuntu– http://howtorapeurjob.tumblr.com/post/140678763346/
• Elasticsearch cluster on Windows– http://howtorapeurjob.tumblr.com/post/140318838221/
• Elasticsearch Cluster Design– http://howtorapeurjob.tumblr.com/post/140438605026/
• CRUD in Elasticsearch– http://howtorapeurjob.tumblr.com/post/140323820711/
• Query DSL in Elasticsearch– http://howtorapeurjob.tumblr.com/post/140616855246/
![Page 18: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/18.jpg)
18
Elasticsearch on Ubuntu
# Close firewallsudo ufw disable
# Edit the configure filesudo gedit /etc/elasticsearch/elasticsearch.yml
12345
# elasticsearch.ymlnetwork.host: host_namecluster.name: "scd500"node.name: "s1"discovery.zen.ping.unicast.hosts: ["host_name"]discovery.zen.ping.multicast.enabled: falsediscovery.zen.minimum_master_nodes: 1node.master: truenode.data: true
123456789
# restart servicesudo service elasticsearch restart
12
• Close Firewall & Edit Configure File
• Configure File
• Restart Service
![Page 19: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/19.jpg)
19
Elasticsearch on Ubuntu
• Install plugin for elasticsearch– Firefox: https://goo.gl/tzwE76 (RESTClient)– Chrome: https://goo.gl/GCc75v (Postman)
check node statushttp://hostname:9200/_cluster/health?pretty
Check Service Statshttp://hostname:9200
![Page 20: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/20.jpg)
20
Elasticsearch cluster on Windows
• Set Java Path on Windows
• Set ConfigurationFile Path: \elasticsearch-2.2.0\config\elasticsearch.yml
# Node S1: \elasticsearch-2.2.0\config\elasticsearch.ymlnetwork.host: ES1 #ES2 #ES3cluster.name: "scd500"node.name: "s1" #"S2" "S3"discovery.zen.ping.unicast.hosts: ["ES1"] discovery.zen.ping.multicast.enabled: falsediscovery.zen.minimum_master_nodes: 1node.master: truenode.data: true
123456789
![Page 21: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/21.jpg)
21
Elasticsearch cluster on Windows
• Setting in elasticsearch.yml– network.host: 要寫可以 ping 到的 hostname ,要記得關防火牆– cluster.name: 一定要同一個名稱,不然就算在同往段裡面也不會視為同群組– node.name: 節點名稱,如果不寫也可以,但是每次重開就會自動重新更新名字– discovery.zen.ping.unicast.hosts: 用來尋找 host 的設定,實際測試在 HTC 網域內設定設定為你的 Master 主機才可以自動加入叢集,倘若在單一電腦上不設定就可以偵測到– discovery.zen.ping.multicast.enabled: 用以防止偵測到預期以外的叢集或節點– node.master: 是否為 Master 節點,若沒指定本身在加入節點的時候就會自動設定– node.data: 是否儲存資料– path.data: Elasticsearch 的資料存放位置– path.logs: Elasticsearch 的 log 存放位置– http.port: 預設的 port 都是 9200
![Page 22: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/22.jpg)
22
Elasticsearch Cluster Design
• Check Cluster Status
– Status: Red -> shard 與副本有問題– Status: yellow -> shard 啟動,副本有問題– Status: Red -> shard 與副本啟動
GET localhost:9200/_cluster/health1
![Page 23: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/23.jpg)
23
Elasticsearch Cluster Design
• Index / Shard / replicas– Elasticsearch 最基本的儲存單位為 shard– 當設定多於一個儲存單位時,會有演算法決定要將資料存在哪個地方– replicas 為你的副本數,在建立 index 的時候就要先決定好你的架設方式
![Page 24: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/24.jpg)
24
Elasticsearch Cluster Design
• Single Shard – 在一台機器裡面只有一個儲存空間,沒有副本
POST localhost:9200/{index_name}{ "settings": { "number_of_shards": 1, "number_of_replicas": 0 }}
1234567
![Page 25: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/25.jpg)
25
Elasticsearch Cluster Design
• 2 Shards – 當有其他 node 被建立起來時,系統會自動將第二個 shard 分散給 Node2
POST localhost:9200/{index_name}{ "settings": { "number_of_shards": 2, "number_of_replicas": 0 }}
1234567
![Page 26: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/26.jpg)
26
Elasticsearch Cluster Design
• 3 Shards , 1 replica– 當有其他 node 被建立起來時,系統會自動將第二個 shard 分散給 Node2s POST localhost:9200/{index_name}
{ "settings": { "number_of_shards": 3, "number_of_replicas": 1 }}
1234567
3 Shards0 Replica1 Node
3 Shards1 Replica3 Nodes
3 Shards1 Replica2 nodes
![Page 27: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/27.jpg)
27
CRUD in Elasticsearch
• REST API 在 Elasticsearch 裡, REST API 結構如下
當執行時, index 與 type 是必要的, id 可以指定或不指定當不指定 id 而塞入資料時,要以 POST 傳遞,不能用 PUT
以設計 Facebook 為例子來看 CRUD 操作每則訊息都包含了以下基本資料
http://localhost:9200/<index>/<type>/[<id>] 1
{ "message" : "", "feel" : "", "location" : "", "datetime" : ""}
123456
![Page 28: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/28.jpg)
28
CRUD in Elasticsearch
• Create Index預設索引為空的,所以要先建立一個為 facebook 的 index
– Linux
– REST Client
– Result
• Query Indices
– Result
curl -XPOST 'http://localhost:9200/facebook'1
{ "acknowledged": true}
123
POST http://localhost:9200/facebook1
GET http://localhost:9200/_cat/indices?v1
health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open facebook 5 1 0 0 795b 795b
12
![Page 29: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/29.jpg)
29
CRUD in Elasticsearch
• Create Document– Linux
– RESR Client
– Result
curl -XPOST 'http://localhost:9200/facebook/yuhsuan_chen' -d ‘{"message": " Hello!!!","feel": "Good","location": "Taoyuan City","datetime": "2016-03-01T12:00:00"}’
1
POST http://localhost:9200/facebook/yuhsuan_chen{ "message": "Hello!!!", "feel": "Good", "location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}
1234567
{ "_index":"facebook", "_type":"yuhsuan_chen", "_id":"AVM7YFHl16LIHfUR_IEh", -> 自動編號或是可以自己指定 "_version":1, -> 修改的第幾個版本 "_shards":{ "total":2, -> 有多少個副本 "successful":1, "failed":0}, "created":true}
123456789101112
![Page 30: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/30.jpg)
30
CRUD in Elasticsearch
• Query Data# 不指定查詢GET http://localhost:9200/_search?
# 指定關鍵字查詢GET http://localhost:9200/_search?q='good'
# 在特定 index 下查詢GET http://localhost:9200/facebook/_search?q='good'
# 在特定 type 下查詢GET http://localhost:9200/facebook/yuhsuan_chen/_search?q='good'
# 尋找 message 這欄位內包含 hello 的資料GET http://localhost:9200/facebook/_search?q=message:hello
# 尋找包含 datetime 這欄位,且其他資料內包含 helloGET http://localhost:9200/facebook/_search?_field=datetime&q=hello
1234567891011121314151617
![Page 31: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/31.jpg)
31
CRUD in Elasticsearch
• Delete Data# 根據 id 刪除資料DELETE http://localhost:9200/facebook/yuhsuan_chen/1
# 刪除 indexDELETE http://localhost:9200/facebook
# 刪除 type在 2.2 版之後已不支援刪除 type ,只能刪除 indexhttps://goo.gl/znbOuB
123456789
![Page 32: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/32.jpg)
32
CRUD in Elasticsearch
• Update Data# 直接更新整個文件POST http://localhost:9200/facebook/yuhsuan_chen/1{ "message": "Hello!!!", "feel": "Good", "location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}
# 更新文件內的某一個資料欄位POST http://localhost:9200/facebook/yuhsuan_chen/1/_update{ "doc": { "message": "haha" }}
12345678910111213141516
![Page 33: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/33.jpg)
33
CRUD in Elasticsearch
• Bulk– 如果 POST link 中已經包含 index 及 type ,在輸入內容的時候可以不用指定– 如果需要自己指定,則需定義 index 及 type– id 可以不指定,但是 delete 的時候還是必須知道 id 才有辦法刪除
要注意很重要的一點,最後結尾一定要加上換行字元 !!!!
# 新增資料POST http://localhost:9200/facebook/yuhsuan_chen/_bulk{"create":{"_id":"1"}}{"message":"Hello!!!","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}{"create":{"_index":"facebook","_type":"Other","_id":"2"}}{"message": "The weather is so good.","feel": "Comfortable","location": "Taoyuan City", "datetime": "2016-03-01T12:01:00"}{"create":{"_id":"3"}}{"message": "I like my job.","feel": "bad","location": "Taoyuan City", "datetime": "2016-03-01T12:02:00"}{"create":{}}{"message": "Time for launch.","feel": "hungry","location": "Taoyuan City", "datetime": "2016-03-01T12:03:00"}
123456789101112131415
![Page 34: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/34.jpg)
34
CRUD in Elasticsearch
• Bulk
要注意很重要的一點,最後結尾一定要加上換行字元 !!!!
顯示剛剛操作的最後結果
# 新增三筆資料,第一筆 location 修改為台北市,刪除第二筆資料POST http://localhost:9200/facebook/_bulk{"create":{"_type":"test","_id":"1"}}{"message":"test1","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}{"create":{"_type":"test","_id":"2"}}{"message":"test2","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}{"create":{"_type":"test","_id":"3"}}{"message":"test3","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-01T12:00:00"}{"update":{"_type":"test","_id":"1"}}{"doc": { "location": "Taipei City"}}{"delete":{"_type":"test","_id":"2"}}
123456789101112131415
GET http://localhost:9200/facebook/_search?q=_type:test1
![Page 35: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/35.jpg)
35
CRUD in Elasticsearch
• Import Data– Syntax
– Example
curl -XPOST localhost:9200/_bulk --data-binary @file.json1
curl -POST "http://yuhsuan_chen_w7p:9200/_bulk" --data-binary @C:\Users\Yuhsuan_chen\Desktop\shakespeare.json1
![Page 36: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/36.jpg)
36
Query DSL in Elasticsearch
• 特別注意如果使用 linux 的 curl套件,可以直接使用 GET 進行參數傳遞如果使用 windows 的 curl套件,會對字串的判別上有問題,尤其是單引號與雙引號,因此不建議使用 windows 版本的 curl。在 windows 上建議直接使用 POSTMAN 進行查詢如果使用套件進行操作,因為大多數的 http 相關的 library 都不支援 GET帶參數,所以建議使用POST 操作IETF 的相關文件內也沒特別規定 GET 是否可以帶參數,但是 Elasticsearch 開發人員覺得 GET比較合理,但實際上查詢時還是使用 POST降低問題發生http://tools.ietf.org/html/rfc7231#section-4.3.1
# Create test dataPOST http://localhost:9200/facebook/yuhsuan_chen/_bulk{"create":{"_id":"1"}}{"message":"Hello!!!","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-01T12:00:00","temp":25.6}{"create":{"_index":"facebook","_type":"Other","_id":"2"}}{"message": "The weather is so good.","feel": "Comfortable","location": "Taipei City", "datetime": "2016-03-01T12:01:00","temp":27.3}{"create":{"_id":"3"}}{"message": "I like my job.","feel": "bad","location": "Taoyuan City", "datetime": "2016-03-01T12:02:00"}{"create":{}}{"message": "Time for launch.","feel": "hungry","location": "Taoyuan City", "datetime": "2016-03-01T12:03:00","temp":30.3}
123456789101112131415
![Page 37: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/37.jpg)
37
Query DSL in Elasticsearch
• Query All Data
• Query Specific Field Data
# Use match_all to query all dataPOST http://localhost:9200/facebook/_search{ "query" : { "match_all" : { } }}
12345678
# Use match to query “city” in location fieldPOST http://localhost:9200/facebook/_search{ "query" : { "match" : { "location" : "city" } }}
123456789
![Page 38: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/38.jpg)
38
Query DSL in Elasticsearch
• Query Multi Data
• Query Value
# Find location/message field include hello/goodPOST http://localhost:9200/facebook/_search { "query" : { "multi_match" : { "query": ["hello","good"], "fields": ["location","message"] } }}
12345678910
# Find the temp >= 27, gt -> 大於 , gte ->大於等於, lt ->小於, lte ->小於等於POST http://localhost:9200/facebook/yuhsuan_chen/_search{ "query" : { "range" : { "temp":{ "gte":27 } } }}
1234567891011
![Page 39: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/39.jpg)
39
Query DSL in Elasticsearch
• Query Data By Regular Expression
• Order Data
# Find the message can be match “fo.*” or “he.*”POST http://localhost:9200/facebook/yuhsuan_chen/_search{ "query" : { "regexp" : { "message":"fo.*|he.*" } }}
123456789
# Query all data, order _type by desc, order _id by ascPOST http://localhost:9200/facebook/yuhsuan_chen/_search { "query" : { "match_all" : {} }, "sort":[ {"_type" : {"order":"desc"}}, {"_id" : {"order":"asc"}} ]}
1234567891011
![Page 40: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/40.jpg)
40
Query DSL in Elasticsearch
• Query Condition– must 代表必須符合– must_not 代表必不能符合– should 可以加重查詢比重– 至少要有一個 should, 但是如果有 must / must_not 就可以不用 should
# Message 中必須包含” .*i.*”, 而且 Message 不含” good”, 如果 feel 中有包含 bad 則加重排名POST http://localhost:9200/facebook/yuhsuan_chen/_search{ "query":{ "bool":{ "must": {"regexp":{"message":".*i.*"}}, "must_not": {"regexp":{"message":"good"}}, "should":{"match":{"feel":"bad"}} } }}
1234567891011
![Page 41: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/41.jpg)
41
Query DSL in Elasticsearch
• Query by “bool”包含 should , _id:3排名會在前面 不包含 should , _id:3排名較低
![Page 42: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/42.jpg)
42
Kibana
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices
• Flexible analytics and visualization platform
• Real-time summary and charting of streaming data
• Intuitive interface for a variety of users
• Instant sharing and embedding of dashboards
• Configure Kibana & Elasticsearch– http://howtorapeurjob.tumblr.com/post/140733691641/
![Page 43: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/43.jpg)
43
Kibana
• Set Configuration
• YML
sudo gedit /opt/kibana/config/kibana.yml1
# \config\kibana.ymlserver.host: "yuhsuan_chen_w7p.htctaoyuan.htc.com.tw"elasticsearch.url: "http://yuhsuan_chen_w7p.htctaoyuan.htc.com.tw:9200"
123
![Page 44: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/44.jpg)
44
Kibana
• Download Sample Fileshttps://www.elastic.co/guide/en/kibana/3.0/snippets/shakespeare.json
https://github.com/bly2k/files/blob/master/accounts.zip?raw=true
https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz
• Create Index# Index: shakespearecurl -XPUT http://localhost:9200/shakespeare{ "mappings" : { "_default_" : { "properties" : { "speaker" : {"type": "string", "index" : "not_analyzed" }, "play_name" : {"type": "string", "index" : "not_analyzed" }, "line_id" : { "type" : "integer" }, "speech_number" : { "type" : "integer" } } } }}
1234567891011121314
![Page 45: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/45.jpg)
45
Kibana
• Create Index# Index: Index: logstash-{datetime} curl -XPUT http://localhost:9200/logstash-2015.05.18 #19 20{ "mappings": { "log": { "properties": { "geo": { "properties": { "coordinates": { "type": "geo_point" } } } } } }}
1234567891011121314151617
![Page 46: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/46.jpg)
46
Kibana
• Import Data
• Check All Indices
• Connect to Kibanahttp://localhost:5601
如果有設定 Nginx 的自動導向可以不用輸入 port剛建立起來第一次會需要一點時間讓 Kibana初始化如果沒有跳到設定就自己點選 setting
# Import Datacurl -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.jsoncurl -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary @shakespeare.jsoncurl -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl
1234
# Check indicescurl 'localhost:9200/_cat/indices?v'
12
![Page 47: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/47.jpg)
47
Kibana
如果有設定 Nginx 的自動導向可以不用輸入 port剛建立起來第一次會需要一點時間讓 Kibana初始化如果沒有跳到設定就自己點選 setting
這裡要建立三個 index 給 Kibana 可以快速搜尋logstash-*ba*shakes*
![Page 48: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/48.jpg)
48
Kibana
這代表著當我們要搜尋 Kibana 時,可以先行過濾要抓取哪個 Elasticsearch 的 index
我們可以看到所有相關的欄位屬性資料,也可以從這邊來做調整回到 Discover 的部分,選擇 ba* 的 index ,搜尋帳號編號 <100 且資產為 >47500 的資料,全部總共有五筆
如果我們想要針對其他欄位做資料選取,可以點選左邊 avaliable fields -> add 就可以看到我們篩選過後的欄位資料
account_number:<100 AND balance:>475001
![Page 49: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/49.jpg)
49
Kibana
• 建立資產年齡分布圓餅圖選擇 Visualize -> Pie chart -> From a new search -> Select an index pattern -> ba*選擇 buckets -> Split Slices -> Aggregation -> Range -> Field -> balance -> 輸入範圍數值
![Page 50: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/50.jpg)
50
Kibana
接著新增一個 Split Slices -> Aggregation -> Terms-> Field -> age -> Order By -> metric: Count -> Order -> Descending -> Size -> 5最後按下上方的綠色三角已產生我們要的圖表資料
如果要儲存可以選擇右上方的儲存
![Page 51: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/51.jpg)
51
Kibana
• 建立 Dashboard選擇以儲存的圖表就會顯示在 Dashboard 上,可以自由調整大小,位置等
![Page 52: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/52.jpg)
52
Nginx
• Nginx port forwarding– http://howtorapeurjob.tumblr.com/post/140665750171/
![Page 53: Introduction to ELK](https://reader033.vdocuments.net/reader033/viewer/2022050914/58ed0fa01a28abce0d8b45c9/html5/thumbnails/53.jpg)
53
ELK parse system log on Ubuntu
• How to set ELK on Ubuntu– http://howtorapeurjob.tumblr.com/post/140837053206/