jvm web application metrics & monitoring · pull /push push server -(process)-> metrics...
TRANSCRIPT
![Page 1: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/1.jpg)
JVM Web Application
Metrics & Monitoring FOLIO
@krrrr38
2
![Page 2: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/2.jpg)
3
![Page 3: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/3.jpg)
1.
2.
3. JVM Web Application
4
![Page 4: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/4.jpg)
5
![Page 5: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/5.jpg)
Metrics
6
![Page 6: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/6.jpg)
Metrics
JVM
http
JMX
jcmd, jstat, ...
7
![Page 7: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/7.jpg)
Metrics
server/process
pull /push
8
![Page 8: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/8.jpg)
pull /pushpush
server -(process)-> metrics storage gateway
e.g. mackerel agent
pull
server <- metrics
e.g. prometheus
9
![Page 9: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/9.jpg)
Example 1mackerel-agent (mackerel-jvm-plugin)
mackerel
https://mackerel.io/
process
10
![Page 10: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/10.jpg)
jvm app (jmx, perf,...)
↓
jstat, jcmd
↓
mackerel-agent
↓
mackerel
↓
mackerel 11
![Page 11: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/11.jpg)
Example 2prometheus, grafana
12
![Page 12: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/12.jpg)
jvm app: /metrics prom metrics
↓
prometheus
↓
grafana
prometheus
13
![Page 13: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/13.jpg)
Example 3telegraf
metrics interface gateway
telegraf
https://github.com/in�uxdata/telegraf
14
![Page 14: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/14.jpg)
15
![Page 15: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/15.jpg)
key value JMX
dropwizard metrics
16
![Page 16: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/16.jpg)
key+labels value
17
![Page 18: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/18.jpg)
response_time_GET_foo = 123msresponse_time_GET_bar = 250msresponse_time_POST_bar = 210msresponse_time_POST_foo_bar = 512ms
19
![Page 19: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/19.jpg)
average/max/95%tile
response_time_GET_foo_average = 123msresponse_time_GET_foo_max = 123msresponse_time_GET_foo_95% = 123msresponse_time_GET_bar_average = 250msresponse_time_GET_bar_max = 250msresponse_time_GET_bar_95% = 250msresponse_time_POST_bar_average = 210msresponse_time_POST_bar_max = 210msresponse_time_POST_bar_95% = 210msresponse_time_POST_foo_bar_average = 512msresponse_time_POST_foo_bar_max = 512msresponse_time_POST_foo_bar_95% = 512ms
95%tile / ...?
20
![Page 20: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/20.jpg)
SELECT * FROM metrics WHERE key IN (response_time_GET_foo_95%,response_time_GET_bar_95%,response_time_POST_bar_95%,response_time_POST_foo_bar_95%)
index
SELECT * FROM metrics WHERE key LIKE 'response_time_%_95\%'
21
![Page 21: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/21.jpg)
response_time{"method"=GET, path=foo, percent=50} = 123msresponse_time{"method"=GET, path=foo, percent=95} = 123msresponse_time{"method"=GET, path=foo, percent=100} = 123ms
template
percent
method=POST
22
![Page 22: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/22.jpg)
23
![Page 23: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/23.jpg)
24
![Page 24: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/24.jpg)
JVM Web ApplicationMetrics
25
![Page 25: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/25.jpg)
heap usage
gc lifecycle
thread pool
connection pool
cpu
request count / response time
queue
...26
![Page 26: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/26.jpg)
heap usage / gc lifecyclememory leak
Xmx
gc
( )
in-memory cache
512M~2G
FullGC STW
gc lifecycle
27
![Page 27: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/27.jpg)
thread / connection poolblocking thread
pool active
e.g. servlet not nio api
jedis/jdbc/... thread pool in application
blocking
thread pool
thread pool
implicit global
default ForkjoinPool
28
![Page 28: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/28.jpg)
server cpu / network / etcapplication
cpu
network
29
![Page 29: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/29.jpg)
request count / responsetime / queue / ...
30
![Page 30: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/30.jpg)
31
![Page 31: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/31.jpg)
micrometer
32
![Page 32: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/32.jpg)
micrometerJava
key+labels value
plugin
JVM System metrics, GC, logback, tomcat,
jetty, hikaricp
jmx, prom, in�ux, datadog agent,...
33
![Page 33: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/33.jpg)
jmx, java agent, -XX:+FlightRecorder , ...
34
![Page 34: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/34.jpg)
agentjava -jar
e.g. NewRelic
35
![Page 35: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/35.jpg)
36
![Page 36: JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage gateway e.g. mackerel agent pull server](https://reader030.vdocuments.net/reader030/viewer/2022040122/5f26fa2f9ede725b1411e3f7/html5/thumbnails/36.jpg)
framegraph:
http://www.brendangregg.com/blog/2015-11-
06/java-mixed-mode-�ame-graphs.html
https://github.com/jvm-pro�ling-tools/perf-
map-agent
https://glowroot.org/
37