what we learned integrating grafana and with prometheus · 2020-05-29 · © 2018 percona. 1...

©2018Percona.

1

DimitriVanoverbeke

What We Learned Integrating Grafana and With PrometheusWhileBuildingPerconaMonitoringandManagement

March2nd,2018GrafanaconEU2018

©2018Percona.

2

Apologiesupfront

©2018Percona.

3

Percona’sPurpose

ToChampionUnbiasedOpenSourceDatabase

SoluDons

©2018Percona.

4

Percona

Wearenotweatherexperts

©2018Percona.

5

Meaning•  WeProvideSoluDons(Support,ConsulDng,Training,Remote

DBA)forOpenSourceDatabases•  AllSoLwareWeReleaseis100%FreeandOpenSource•  HelpingCustomerstoAvoidSoLwareVendorLock-In•  ProvidingOpenSourceAlternaDvestothe“Enterprise”

SoLwareediDons

©2018Percona.

6

•  Commercial•  CloudOnly•  DoItYourself

•  MySQLEnterpriseMonitor•  MongoDBOpsManager

StateofOpenSourceDatabaseMonitoring

©2018Percona.

7

•  100%FreeandOpenSource•  WorksInandoutofthe

Cloud•  Deployin15minutes•  PurposeBuiltforOpen

SourceDatabaseMonitoring•  ForDevelopersandDBAs

PerconaMonitoringandManagement

©2018Percona.

8

TheMainQuesDonHowtobuildGreatSoLwarewithSmallTeam?

©2018Percona.

9

Youcandoit!•  WorkingtogetherwithOpenSourceCommunity

©2018Percona.

10

Wechose

©2018Percona.

11

Reasons•  LargeCommunity•  DesignedasanExtensiblePlacorm•  VeryResponsiveDevelopmentTeam•  PermissiveLicense•  MembersofCloudNaDveCompuDngFoundaDon•  ModernTechnologySupport

©2018Percona.

12

OurInnovaDonFocus•  QueriesarenotonlyMetrics•  SimplifyDeployment(easydeployment)•  PowerfulDashboards

©2018Percona.

13

WhatQueriesareCausingtheLoad

©2018Percona.

14

WhyAretheyCausingthisLoad

©2018Percona.

15

HowtoImprovetheirPerformance

©2018Percona.

16

IntegraDngwithGrafana•  UsingGreatlyDesignedGrafanaDateRangeSelector•  GrafanaHostSelector•  GrafanaCross-DashboardLinks•  IntegraDonDonethroughIFRAME

©2018Percona.

17

SimplifyDeployment•  DeployingMulDplePrometheusExportersisHard•  CustomAgentisusedforQueryFuncDons•  Packageeverythingtosingle“agent”package•  pmm-adminaddmysql

©2018Percona.

18

IntegraDonwithCloud(AWS)•  AutomaDcallySearchandDiscoverSupportedInstances•  AnotherCustomGrafanaPanel

©2018Percona.

19

PowerfulDashboards

©2018Percona.

20

DashboardLearningExperience•  GreatDashboardAPI•  FantasDcsupportforDashboardVersioning•  CanuseIDfordashboardidenDficaDon,notname•  FantasDcTemplaDngEngine•  UsingBothRepeaDngrowsandRepeatedColumns

©2018Percona.

21

UsingRepeatedRows

©2018Percona.

22

UsingRepeatedPanels

©2018Percona.

23

UseTagstobuildoutNavigaDonMenu

©2018Percona.

24

Challenges

©2018Percona.

25

AlerDng•  HowtouseAlerDnginmulD-serverenvironments•  WanttoalertonServiceratherthanindividualserver•  Want“visual”alertsintegratedwithGrafana•  UseGrafanaAlertsorbuildappforPrometheusAlerts?

©2018Percona.

26

AnnotaDons•  WantannotaDonstobeenteredbyuser•  Butwantthemtoappearonthe“similar”panelseverywhere•  Howtodefinesuch“similar”panels?

©2018Percona.

27

SingleStat•  NeedfixedYmaxvalueforSinglestatdashboardtobeher

visualizepercents•  Lookingtocontributepatch

©2018Percona.

28

DynamicResoluDon•  WehavedifferentdatascrapedwithdifferentresoluDon•  Somedatais1secondresoluDon(statusvariables)•  Otheris60secresoluDon(variables,tableinformaDon)•  UsercanchoosetheirownresoluDonbasedonoverhead•  Howtoalignitonthesamegraphs?•  Makedatatotallyzoomable

©2018Percona.

29

DynamicResoluDon•  GrafanaSupports“minimumresoluDon”

•  HardtomanagefordynamicuserconfiguraDon•  PrometheusFuncDonsareverystrict

•  “donotinventanydata•  rate()[1s]WillreturnNULLnotfinding2datapoints

•  Useuglyhacksofar•  Rate()[$interval]orirate()[5m]•  GivesbestavailableresoluDon

©2018Percona.

30

BeherselecDonforAutoInterval•  AutoResoluDoniscomputed•  Intervalselectedmaybe“onedatapointfor2.2minutes”•  Wouldbegreattosetittofixedsetofintervals•  1s;5s;1min;5min;1h…

©2018Percona.

31

SAVE THE DATE!

Learn More about Using PMM in Practice!

www.perconalive.com

April 23-25, 2018 Santa Clara Convention Center

©2018Percona.

32

ThankYou!