zabbix network monitoring essentials - omid-online.com · table of contents zabbix network...
TRANSCRIPT
TableofContents
ZabbixNetworkMonitoringEssentials
Credits
AbouttheAuthors
AbouttheReviewers
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmore
Whysubscribe?
FreeaccessforPacktaccountholders
Preface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Errata
Piracy
Questions
1.InstallingaDistributedZabbixSetup
Zabbixarchitectures
UnderstandingZabbixdataflow
UnderstandingtheZabbixproxies’dataflow
InstallingZabbix
Installingfrompackages
SettingupaZabbixagent
CreatingaZabbixagentpackagewithCheckInstall
Serverconfiguration
Installingadatabase
www.it-ebooks.info
Consideringthedatabasesize
MySQLpartitioning
InstallingaZabbixproxy
InstallingtheWebGUIinterface
Summary
2.ActiveMonitoringofYourDevices
UnderstandingZabbixhosts
Hostsandhostgroups
Hostinterfaces
Hostinventory
GoingbeyondZabbixagents
Simplechecks
KeepingSNMPsimple
GettingSNMPdataintoZabbix
FindingtherightOIDstomonitor
MappingSNMPOIDstoZabbixitems
Gettingdatatypesright
SNMPtraps
Snmptrapd
TransformingatrapintoaZabbixitem
Gettingnetflowfromthedevicestothemonitoringserver
Receivingnetflowdataonyourserver
MonitoringalogfilewithZabbix
Summary
3.MonitoringYourNetworkServices
MonitoringtheDNS
DNS–responsetime
DNSSEC–monitoringthezonerollover
Apachemonitoring
NTPmonitoring
NTP–whatarewemonitoring?
www.it-ebooks.info
Squidmonitoring
Summary
4.DiscoveringYourNetwork
FindinghoststheZabbixway
Definingactionconditions
Choosingactionoperations
Remotecommands
Low-leveldiscovery
Summary
5.VisualizingYourTopologywithMapsandGraphs
Creatingcustomgraphs
Maps–aquicksetupforalargetopology
Maps–automatingtheDOTcreation
DraftingZabbixmapsfromDOT
Puttingeverythingtogetherwithscreens
Summary
A.PartitioningtheZabbixDatabase
MySQLpartitioning
Thepartition_maintenanceprocedure
Thepartition_createprocedure
Thepartition_verifyprocedure
Thepartition_dropprocedure
Thepartition_maintenance_allprocedure
Housekeepingconfiguration
B.CollectingSquidMetrics
Squidmetricscript
Index
www.it-ebooks.info
ZabbixNetworkMonitoringEssentialsCopyright©2015PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthors,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:February2015
Productionreference:1210215
PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78439-976-4
www.packtpub.com
www.it-ebooks.info
CreditsAuthors
AndreaDalleVacche
StefanoKewanLee
Reviewers
RaviBhure
NicholasPier
NicolaVolpini
CommissioningEditor
AmarabhaBanerjee
AcquisitionEditor
NikhilKarkal
ContentDevelopmentEditor
SiddheshSalvi
TechnicalEditor
HumeraShaikh
CopyEditor
SarangChari
ProjectCoordinator
KrantiBerde
Proofreaders
SimranBhogal
LindaMorris
Indexer
HemanginiBari
Graphics
DishaHaria
ProductionCoordinator
AparnaBhagat
CoverWork
AparnaBhagat
www.it-ebooks.info
AbouttheAuthorsAndreaDalleVaccheisahighlyskilledITprofessionalwithover14yearsofexperienceintheITindustryandbanking.HegraduatedfromUniversitàdegliStudidiFerrarawithaninformationtechnologycertification.ThislaidthetechnologyfoundationthatAndreahasbuiltoneversince.Andreahasacquiredvariousindustry-respectedaccreditations,whichincludeCisco,Oracle,RHCE,ITIL,andofcourse,Zabbix.Throughouthiscareer,hehasworkedinmanylarge-scaleenvironments,ofteninrolesthathavebeenverycomplex,onaconsultantbasis.Thishasfurtherenhancedhisgrowingskillset,addingtohispracticalknowledgebaseandincreasinghisappetitefortheoreticaltechnicalstudying.
Andrea’sloveforZabbixcamefromhistimespentintheOracleworldasadatabaseadministrator/developer.Histimewasspentmainlyonreducingownershipcosts,specializinginmonitoringandautomation.ThisiswherehecameacrossZabbixandtheflexibilityitoffered,bothtechnicallyandadministratively.Withthisasalaunchpad,AndreawasinspiredtodevelopOrabbix,thefirstopensourcesoftwaretomonitorOracle’scompleteintegrationwithZabbix.HehaspublishedanumberofarticlesonZabbix-relatedsoftware,suchasDBforBIX.Hisprojectsarepubliclyavailableathttp://www.smartmarmot.com.Currently,Andreaisworkingasaseniorarchitectforaleadingglobalinvestmentbankinaverydiverseandchallengingenvironment.HedealswithmanyaspectsoftheUnix/Linuxplatformsaswellasmanytypesofthird-partysoftware,whicharestrategicallyalignedtothebank’stechnicalroadmap.Inadditiontothistitle,AndreaDalleVaccheisacoauthorofMasteringZabbix,PacktPublishing.
StefanoKewanLeeisanITconsultantwithmorethan12yearsofexperienceinsystemintegration,security,andadministration.HeisacertifiedZabbixspecialistinlargeenvironmentsholdsaLinuxadministrationcertificationfromtheLPIandaGIACGCFWcertificationfromSANSInstitute.Whenhe’snotbusybreakingwebsites,helivesinthecountrysidewithhistwocatsandtwodogsandpracticesmartialarts.Inadditiontothistitle,StefanoKewanLeeisacoauthorofMasteringZabbix,PacktPublishing.
www.it-ebooks.info
AbouttheReviewersRaviBhureisbasicallyanITengineerwithnicheskills,suchasChef,CloudAnsible,SaltStack,Python,Ruby,andShell/Bash.Healsowritescodeforinfrastructure,dailyIToperations,andsoon.Inshort,heisfondofusinghisskillsandknowledgeoffault-tolerantsolutionsfortheday-to-daymaintenanceofmission-criticalproductioninfrastructure.
Ravistartedinteractingwithcomputerssince1996whenhegothisfirstcomputerathome.Thingschangedveryfast,andin1998,heenteredthemagicalworldoftheInternet☺forthefirsttimeever,whichchangedhislife!Hestartedhisowncybercafein1999.In2004,hegothisfirstjobasafieldengineer,hiredtomaintainandsupportVRIUFOsystems.After2years,hemovedtoPuneandworkedwithmanyorganizations,suchasVyomLabs,GlamIndia,Symphony,andDhingana.
Themosthappeningandinterestingfactabouthisdiverseexposureisthatheisfromanartsbackground.Yes,heholdsabachelor’sdegreeinartsfromSRTMUniversity,Nanded,Maharashtra,India.Andweallwillhavetoagreethathehasthearttosolveproblems☺,agreatinspirationforpeoplewhoarenonengineers!
Currently,RaviisassociatedwithOpexSoftwareasaseniorDevOpsengineer.
NicholasPierisanetworkengineerinthemanagedservices/professionalservicesfield.HisexperienceincludesdesigningdatacenternetworkinfrastructureswithvirtualizationandSANsolutions,webdevelopment,andwritingmiddlewareforbusinessapplications.Atthetimeofwritingthis,Nicholasholdsanumberofindustrycertifications,includingtheCiscoCCNP,VMwareVCP5-DCV,andvariousotherCiscoandCompTIAcertifications.Inhisfreetime,heindulgesinhispassionforcraftbeer,distancerunning,andreading.
I’dliketothankPacktPublishingforthisopportunity!
NicolaVolpinihasbeenplayingwithtechnologyfromayoungage,havingahardtimeresistingtheurgetodisassemblecomplextoysorkitchenappliances.
Theloveforcomputersoriginatedaroundhistenthbirthday,whenheaccidentallytoastedhisfirstCPU.Thisepisodeonlyincreasedhisfascinationforcomputers,andtheaccidents,fortunately,stopped.
Forthepast10years,he’sbeenworkingasanITprofessional,specializinginenterprisenetworkingandsystemadministration.ExperimentingwiththemostdiversetechnologiesinthefieldandbeinganavidfanoftheFOSSphilosophy,Linux,and*BSD,hedreamsofseeingthecollaborativethinkingoftheFOSSmovementhelpinspiretheworld.
He’scurrentlyworkingatStockholm,Sweden,whereheresideswithhisgirlfriend.
www.it-ebooks.info
Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
https://www2.packtpub.com/books/subscription/packtlib
DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.
www.it-ebooks.info
Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser
www.it-ebooks.info
FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.
www.it-ebooks.info
PrefaceNetworkadministratorsarefacinganinterestingchallengethesedays.Ontheonehand,computernetworksarenotsomethingnewanymore.Theyhavebeenaroundforquiteawhile:theirphysicalcomponentsandcommunicationprotocolsarefairlywellunderstoodanddon’trepresentabigmysterytoanincreasingnumberofprofessionals.Moreover,networkappliancesaregettingcheaperandeasiertosetup,tothepointthatitdoesn’ttakeacertifiedspecialisttoinstallandconfigureasimplenetworkorconnectittoothernetworks.Theveryconceptofnetworkingissowidespreadandingrainedinhowusersanddevelopersthinkofacomputersystemthatbeingonlineinsomeformisexpectedandtakenforgranted.Inotherwords,acomputernetworkisincreasinglyseenasacommodity.
Ontheotherhand,theverysameforcesthatarecallingforsimpler,easier,accessiblenetworksaretheonesthatareactuallypushingthemtogrowmoreandmorecomplexeveryday.It’samatterofbothquantityandquality.Thenumberofconnecteddevicesonagivennetworkisalmostalwaysconstantlygrowingandsoistheamountofdataexchanged:mediastreams,applicationdata,backups,databasequeries,andreplicationtendtosaturatebandwidthjustasmuchastheyeatupstoragespace.Asforquality,therearedozensofdifferentrequirementsthatfactorinagivennetworksetup:fromhavingtomanagedifferentphysicalmediums(fiber,cable,radio,andsoon),totheneedtoprovidehighperformanceandavailability,bothontheconnectionandontheapplicationlevel;fromtheneedtoincreaseperformanceandreliabilityforgeographicallinks,toprovidingconfidentiality,security,anddataintegrityatalllevels,andthelistgoeson.
Thesetwocontrasting,yetintertwined,tendenciesareforcingnetworkadministratorstodomore(moreservices,moreavailability,andmoreperformance)withless(lessbudget,butalsolessattentionfromthemanagementcomparedtonewer,flashiertechnologies).Now,morethanever,asanetworkadmin,youneedtobeabletokeepaneyeonyournetworkinordertokeepitinahealthystate,butalsotoquicklyidentifyandresolvebottlenecksandoutagesofanykind—orbetteryet,findwaystoanticipateandworkaroundthembeforetheyhappen.You’llalsoneedtointegrateyoursystemswithdifferenttoolsandenvironments(bothlegacyandstrategicones)thatwillbeoutofyourdirectcontrol,suchasassetdatabases,incidentmanagementsystems,accountingandprofilingsystems,andsoon.Evenmoreimportantly,you’llneedtobeabletoshowyourworkandexplainyourneedsinclear,understandabletermstonontechnicalpeople.
Now,ifweweretosaythatZabbixistheperfect,one-size-fits-allsolutiontoallyournetworkmonitoringandmanagementproblems,wewouldclearlybelying.Tothisday,nosuchtoolexistsdespitewhatmanyvendorswantyoutobelieve.Eveniftheyhavemanyfeaturesincommon,whenitcomestomonitoringandcapacitymanagement,everynetworkhasitsownquirks,specialcases,andpeculiarneeds,tothepointthatanytoolhastobecarefullytunedtotheenvironmentorfacetheriskofbecominguselessandneglectedveryquickly.
WhatistrueisthatZabbixisamonitoringsystempowerfulenoughandflexibleenough
www.it-ebooks.info
that,withtherightamountofwork,canbecustomizedtomeetyourspecificneeds.Andagain,thoseneedsarenotlimitedtomonitoringandalerting,butalsotoperformanceanalysisandprediction,SLAreporting,andsoon.WhenusingZabbixtomonitoranenvironment,youcancertainlycreateitemsthatrepresentvitalmetricsforthenetworkinordertohaveareal-timepictureofwhat’shappening.However,thosesameitemscanalsoproveveryusefultoanalyzeperformancebottlenecksandtoplannetworkexpansionandevolution.Items,triggers,andactionscanworktogethertoletyoutakeanactiveroleinmonitoringyournetworkandeasilyidentifyandpre-emptcriticaloutages.
Inthisbook,we’llassumethatyoualreadyknowZabbixasageneral-purposemonitoringtool,andthatyoualsousedittoacertainextent.Specifically,wewon’tcovertopicssuchasitem,trigger,oractioncreationandconfigurationwithabasic,step-by-stepapproach.Here,wewanttofocusonafewtopicsthatcouldbeofparticularinterestfornetworkadministrators,andwe’lltrytohelpthemfindtheirownanswerstoreal-worldquestionssuchasthefollowing:
Ihavealargenumberofappliancestomonitorandhavetokeepmonitoringdataavailableforalongtimeduetoregulatoryrequirements.HowdoIinstallandconfigureZabbixsothatitisabletomanageeffectivelythislargeamountofdata?Whatarethebestmetricstocollectinordertobothhaveaneffectivereal-timemonitoringsolutionandleveragehistoricaldatatomakeperformanceanalysisandpredictions?ManyZabbixguidesandtutorialsfocusonusingtheZabbixagent.Theagentiscertainlypowerfulanduseful,buthowdoIleverageinaneffectiveandsecurewaymonitoringprotocolsthatarealreadyavailableonmynetwork,suchasSNMPandnetflow?Loadbalancers,proxies,andwebserverssometimesfallunderagrayareabetweennetworkandapplicationadministration.Ihaveabunchofwebserversandproxiestomonitor.Whatkindofmetricsaremostusefultocheck?Ihaveacomplexnetworkwithhoststhataredeployedanddecommissionedonadailybasis.HowdoIkeepmymonitoringsolutionup-to-datewithoutresortingtolong,error-pronemanualinterventionsasmuchaspossible?NowthatIhavecollectedalargeamountofmonitoringandperformancedata,howcanIanalyzeitandshowtheresultsinameaningfulway?HowdoIputtogetherthegraphsIhaveavailabletoshowhowtheyarerelated?
Inthecourseofthenextfewchapters,we’lltrytoprovidesomepointersonhowtoanswerthosequestions.Wediscussasmanypracticalexamplesandreal-worldapplicationsaswecanaroundthesubjectofnetworkmonitoring,butmorethananything,wewantedtoshowyouhowit’srelativelysimpletoleverageZabbix’spowerandflexibilitytoyourownneeds.
Theaimofthisbookisnottoprovideyouwithasetofprepackagedrecipesandsolutionsthatyoucanapplyuncriticallytoyourownenvironment.Eventhoughweprovidedsomescriptsandcodethataretestedandworking(andhopefullyyou’llfindthemuseful),therealintentionwasalwaystogiveyouadeeperunderstandingofthewayZabbixworksso
www.it-ebooks.info
thatyouareabletocreateyourownsolutionstoyourownchallenges.
Wehopewehavesucceededinourgoal,andthatbytheendofthebook,you’llfindyourselfamoreconfidentnetworkadministratorandamoreproficientZabbixuser.Evenifthiswillnotbethecase,wehopeyou’llbeabletofindsomethingusefulinthefollowingchapters:wetouchupondifferentaspectsofZabbixandnetworkmonitoringandalsodiscussacoupleoflessknownfeaturesthatyoumightfindveryinterestingnonetheless.
So,withoutfurtherado,let’sgetstartedwiththeactualcontentwewanttoshowyou.
www.it-ebooks.info
WhatthisbookcoversChapter1,InstallingaDistributedZabbixSetup,teachesyouhowtoinstallZabbixinadistributedsetup,withalargeuseofproxies.Thechapterwillguideyouthroughallthepossiblesetupscenarios,showingyouthemaindifferencesbetweentheactiveandpassiveproxysetup.ThischapterwillexplainhowtoprepareandsetupaZabbixinstallation,whichisreadytobegrownwithinyourinfrastructure,readytosupportyou,andmonitoralargeenvironmentorevenaverylargeone.
Chapter2,ActiveMonitoringofYourDevices,offersyouafewveryusefulexamplesofthedifferentmonitoringpossibilitiesZabbixcanachievebyrelyingondifferentmethodsandprotocols.You’llseehowtoqueryyournetworkfromthelinkleveluptoroutingandnetworkflowusingICMP,SNMP,andlog-parsingfacilitiestocollectyourmeasurements.Youwillalsolearnhowtoextractmeaningfulinformationfromthegathereddatausingaggregatedandcalculateditems,andconfiguringcomplextriggersthatwillalertyouaboutrealnetworkissueswhileminimizingsignalnoiseandfalsepositives.
Chapter3,MonitoringYourNetworkServices,takesyouthroughhowtoeffectivelymonitorthemostcriticalnetworkservices,suchasDNS,DHCP,NTP,Apacheproxy/reverseproxies,andproxycacheSquid.Asitiseasytounderstand,allofthemarecriticalserviceswhereasimpleissuecanaffectyournetworksetupandquicklypropagatetheissuetoyourentirenetwork.Youwillunderstandhowtoextractmeaningfulmetricsandusefuldatafromallthelistedservices,beingablethennotonlytomonitortheirownreliability,butalsotoacquireimportantmetricsthatcanhelpyoutopredictfailuresorissues.
Chapter4,DiscoveringYourNetwork,explainshowtodeeplyautomatethemonitoringconfigurationofnetworkobjects.Itwillmassivelyusethebuilt-indiscoveryfeatureinordertokeepthemonitoringsolutionup-to-datewithinanevolvingnetworkenvironment.ThischapterisdividedintotwocorepartsthatcoverthetwomainlevelsofZabbix’sdiscovery:hostdiscoveryandlow-leveldiscovery.
Chapter5,VisualizingYourTopologywithMapsandGraphs,showsyouhowtocreatecomplexgraphsfromyouritem’snumericalvalues,automaticallydrawmapsthatreflectthecurrentstatusofyournetwork,andbringitalltogetherusingscreensasatooltocustomizemonitoringdatapresentation.ThischapteralsopresentsasmartwaytoautomatetheinitialstartupofyourZabbix’ssetup,makingyouabletodrawnetworkdiagramsusingmapsinafullyautomatedway.Youwillthenlearnaproduction-readymethodtomaintainmapswhileyournetworkisgrowingorrapidlychanging.
AppendixA,PartitioningtheZabbixDatabase,containsalltherequiredsoftwareandstoredprocedurestoefficientlypartitionyourZabbixdatabase.
AppendixB,CollectingSquidMetrics,containsthesoftwareusedtomonitorSquid.
www.it-ebooks.info
WhatyouneedforthisbookThesoftwarethathasbeenusedandisnecessaryforthisbookis:
LinuxRedHatEnterpriseLinux6.5orhigherZabbix4.2ApacheHTTPD2.2MySQLServer-5.1Netflow1.6.12Nmap
Thisbookalsorequiresanintermediateexperienceinshellscripting,abasic-to-intermediateknowledgeofPython,andanintermediateknowledgeofZabbix.
Anyway,alltheexamplesdiscussedandproposedinthisbookareexplainedwellandcommentedupon.Thesameapproachhasbeenappliedeventothesoftwareusedonthisbookwhereitisexplained,withareasonablelevelofdetail,howtosetupandconfigureeachsoftwarecomponent.
www.it-ebooks.info
WhothisbookisforThisbookisintendedforexperiencednetworkadministratorslookingforacomprehensivemonitoringsolutionfortheirnetworks.ThereadermusthaveagoodknowledgeofUnix/Linux,networkingconcepts,protocols,andappliancesandabasic-to-intermediateknowledgeofZabbix.Thereaderwillbeguidedstepbysteptomanageandleadalltheimportantpointsyouwillhavetodealwith.Youwillthenbeabletostartupaneffectiveandlarge-environment-readyZabbixmonitoringsolutionthatwillbeaperfectfitwithinyournetwork.
www.it-ebooks.info
ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.
Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“OntheZabbixserver-side,youneedtocarefullysetthevalueofStartTrappers=.”
Ablockofcodeissetasfollows:
#FirstofallweneedtoimportcsvandNetworkx
importcsv
importnetworkxasnx
#Thenweneedtodefinewhoisourzabbixserverandsomeotherdetailto
properlyproducetheDOTfile
zabbix_service_ipaddr="192.168.1.100"
main_loop_ipaddr="10.12.20.1"
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:
#wecanopenourCSVfile
csv_reader=csv.DictReader(open('my_export.csv'),\
delimiter=",",\
fieldnames=("ipaddress","hostname","oid","dontcare","neighbors"))
#Skiptheheader
csv_reader.next()
Anycommand-lineinputoroutputiswrittenasfollows:
#chkconfig--level345zabbix-serveron
Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“Thereisaclearwarningonthewebsitethatwarnsuswiththisstatement:TheApplianceisnotintendedforseriousproductionuseatthistime.”
NoteWarningsorimportantnotesappearinaboxlikethis.
TipTipsandtricksappearlikethis.
www.it-ebooks.info
ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.
Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.
Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.
www.it-ebooks.info
CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.
www.it-ebooks.info
DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
www.it-ebooks.info
ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.
Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.
www.it-ebooks.info
PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.
Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.
Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.
www.it-ebooks.info
QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.
www.it-ebooks.info
Chapter1.InstallingaDistributedZabbixSetupMostlikely,ifyouarereadingthisbook,youhavealreadyusedandinstalledZabbixasanetworkmonitoringsolution.Now,inthischapter,wewillseehowtoinstallZabbixinadistributedsetup,eventuallymovingontoalargeuseofproxies.Thechapterwilltakeyouthroughallthepossiblescenariosandexplainthemaindifferencesbetweentheactiveandpassiveproxysetup.Usually,thefirstZabbixinstallationisdoneasapartoftheconcepttoseewhethertheplatformisgoodenoughforyou.Here,thecommonerroristostartusingthissetuponalargeproductionenvironment.Afterreadingthischapter,youwillbereadytoinstallandsetupalargeenvironmentreadyinfrastructure.
Inthischapter,wewillexplainhowtoprepareandsetupaZabbixinstallation,whichisreadytobegrownwithinyourinfrastructure,andreadyforalargetoaverylargeenvironment.ThisbookismainlyfocusedonZabbixfornetworkmonitoring.Thischapterwillquicklytakeyouthroughtheinstallationprocess,emphasizingonallthemostimportantpointsyouneedtoconsider.Inthenextchapter,wewillspendmoretimedescribingabetterapproachtomonitoryournetworkdevicesandhowtoretrieveallthecriticalmetricsfromthem.Afterreadingthischapter,youwillbecomeawareofthecommunicationbetweenserverandproxiesbeingabletomixtheactiveandpassivesetupinordertoimproveyourinfrastructure.YoucanextendthestrongcentralZabbixcoresetupwithmanylightweightandeffectiveZabbixproxiesactingasasatelliteinsideyournetworktoimproveyourmonitoringsystem.
www.it-ebooks.info
ZabbixarchitecturesZabbixwasbornasadistributednetworkmonitoringtoolwithacentralwebinterfacewhereyoucanmanagealmosteverything.Nowadays,withZabbix2.4,thenumberofpossiblearchitectureshasbeenreducedtoasingleserversetupandaZabbix-proxiesdistributedsetup.
NoteFromZabbix2.4,thenode-setupwasdiscontinued.Moreinformationisavailableathttps://www.zabbix.com/documentation/2.4/manual/introduction/whatsnew240#node-based_distributed_monitoring_removed.
Now,thesimplestarchitecture(whichisreadytohandlelargeenvironmentssuccessfully)thatyoucanimplementcomposesofthreeservers:
WebserverRDBMSserverZabbixserver
Topreparethissimplesetupforalargeenvironmentsetting,it’sbettertouseadedicatedserverforeachoneofthesecomponents.
Thisisthesimplestsetupthatcanbeeasilyextendedandisreadytosupportalargeenvironment.
Theproposedarchitectureisshowninthefollowingdiagram:
ThiskindofsetupcanbeextendedbyaddingmanyZabbixproxiesresultinginaproxy-basedsetup.Theproxy-basedsetupisimplementedwithoneZabbixserverandseveralproxies:oneproxyperbranch,datacenteror,inourcase,foreachremotenetworksegmentyouneedtomonitor.
Thisconfigurationiseasytomaintainandofferstheadvantagetohaveacentralizedmonitoringsolution.Thiskindofconfigurationistherightbalancebetweenlargeenvironmentmonitoringandcomplexity.
www.it-ebooks.info
TheZabbixproxy,likeaserver,isusedtocollectdatafromanynumberofhostsordevices,acquiringallthemetricsrequestedandactingasaproxy.Thismeansthatitcanretainthisdataforanarbitraryperiodoftime,relyingonadedicateddatabasetodoso.Theproxydoesn’thaveafrontendandismanageddirectlyfromthecentralserver.
NoteTheproxylimitsitselftodatacollectionwithouttriggerevaluationsoractions;allthedataisstoredinitsdatabase.Forthisreason,it’sbettertouseanefficientrobustRDBMSthatcanpreventdatalossincaseofacrash.
AllthesecharacteristicsmaketheZabbixproxyalightweighttooltodeployandoffloadsomechecksfromthecentralserver.Ourobjectiveistocontrolandstreamlinetheflowofmonitoreddataacrossnetworks,andtheZabbixproxygivesusthepossibilitytosplitandsegregateitemsanddataonthedifferentnetworks.Themostimportantfeatureisthattheacquiredmetricsarestoredinitsdatabase.Therefore,incaseofanetworkloss,youwillnotlosethem.
www.it-ebooks.info
UnderstandingZabbixdataflowThestandardZabbixdataflowiscomposedofseveralactorsthatsenddatatoourZabbixserver.OfallthesourcesthatcansenddatatoourZabbixserver,wecanidentifythreemaindatasources:
ZabbixagentZabbixsenderOtheragents(externalscriptsorcomponentsbuiltinhouse)
Theotheragentsrepresentedinthenextdiagramcanbeoftwomaintypes:
Customand/orthird-partyagentsZabbixproxy
Asthediagramdisplaysthedatathatgetsacquiredfrommanydifferentsourcesintheformofitems.Attheendofthediagram,youseetheGUI,whichpracticallyrepresentstheusersconnectedandthedatabasethatistheplacewhereallthevaluesarestored.
Inthenextsection,wewilldivedeepintotheZabbixproxies’dataflow.
www.it-ebooks.info
UnderstandingtheZabbixproxies’dataflowZabbixproxiescanoperateintwodifferentmodes,activeandpassive.Thedefaultsetupistheactiveproxy.Inthissetup,theproxyinitiatesallconnectionstotheZabbixserver,theoneusedtoretrieveconfigurationinformationonmonitoredobjects,andtheconnectiontosendmeasurementsbacktotheserver.Here,youcanchangeandtweakthefrequencyofthesetwoactivitiesbysettingthefollowingvariablesintheproxyconfigurationfile:/etc/zabbix/zabbix_proxy.conf:
ConfigFrequency=3600
DataSenderFrequency=1
Valuesareexpressedinseconds.OntheZabbixserver-side,youneedtocarefullysetthevalueofStartTrappers=.
Thisvalueneedstobegreaterthanthenumberofallactiveproxiesandnodesyoudeployed.Thetrapperprocesses,indeed,managealltheincominginformationfromtheproxies.
NotePleasenotethattheserverwillforkextraprocessesasrequired,ifneeded,butitisstronglyadvisabletopreforkalltheprocessesthatareneededduringthestartup.Thiswillreducetheoverheadduringthenormaloperation.
Ontheproxyside,anotherparametertoconsideris:
HeartbeatFrequency
Thisparametersetsasortofkeepalive,whichafterthedefinednumberofseconds,willcontacttheserveralthoughitdoesn’thaveanydatatosend.Theproxyavailabilitycanbeeasilycheckedwiththefollowingitem:
zabbix[proxy,"proxyuniquename",lastaccess]
Heretheproxyuniquename,ofcourse,istheidentifieryouassignedtotheproxyduringdeployment.Theitemwillreturnthenumberofsecondsasthelasttimethattheproxywascontacted,avalueyoucanthenusewiththeappropriatetriggeringfunctions.
TipIt’sreallyimportanttohaveatriggerassociatedtothisitem,soyoucanbewarnedincaseofconnectionloss.Lookingatthetrendofthistrigger,youcanlearnaboutaneventualreapingtimesetonthefirewall.Let’slookatapracticalexample:ifyounoticethatafter5minutesyourconnectionsaredropped,settheheartbeatfrequencyto120secondsandcheckforthelastaccesstimeabove300seconds.
Inthefollowingdiagram,youcanseethecommunicationflowbetweentheZabbixserverandtheproxy:
www.it-ebooks.info
Asyoucanseefromthediagram,theserverwillwaittoreceiverequestsfromtheproxyandnothingmore.
NoteTheactiveproxyisthemostefficientwaytooffloaddutiesfromtheserver.Indeed,theserverwilljustsitherewaitingtobeaskedaboutchangesinconfiguration,ortoreceivenewmonitoringdata.
Ontheotherside,proxiesareusuallydeployedtomonitorsecurenetworksegmentswithstrictoutgoingtrafficpolicies,andareusuallyinstalledonDMZs.Inthesekindofscenarios,normally,itisverydifficulttoobtainpermissionfortheproxytoinitiatethecommunicationwiththeserver.Unfortunately,it’snotjustduetopolicies.DMZsareisolatedasmuchaspossiblefrominternalnetworks,astheyneedtobeassecureastheycan.Generally,it’softeneasierandmoreacceptedfromasecuritypointofviewtoinitiateaconnectionfromtheinternalnetworktoaDMZ.Inthiskindofscenario,thepassiveproxyisveryhelpful.Thepassiveproxyisalmostamirroredimageoftheactiveproxysetup,asyoucanseeinthefollowingdiagram:
Withthisconfiguration,theZabbixserverwillcontacttheproxyperiodicallytodelivertheconfigurationchangesandtorequesttheitemvaluestheproxyisholding.
Thisistheproxyconfigurationtoenabletheproxyyouneedtoset:
www.it-ebooks.info
ProxyMode=1
Thisparameterspecifiesthepassiveproxy,youdon’tneedtodoanythingelse.Now,ontheserverside,youneedtosetthefollowingparameters:
StartProxyPollers=
Thiswillsetthenumberofprocessesdedicatedtothepassiveproxies
NoteTheStartProxyPollersparametershouldmatchthenumberofpassiveproxiesyouhavedeployed.
ProxyConfigFrequency=
Thisvalueexpressesthefrequencywithwhichtheserversendstheconfigurationtoitsproxy
ProxyDataFrequency=
Thisistheintervalparameterthatexpressesthenumberofsecondsbetweentwoconsecutiverequeststogettheacquiredmetricsfromtheproxy
Theitemusedtocheckapassiveproxy’savailabilityisasfollows:
zabbix[proxy,"proxyuniquename",lastaccess]
Thisisexactlythesameastheactiveone.
Thepassiveproxyenablesustogathermonitoringdatafromotherwiseclosedandlockeddownnetworkswithaslightlyincreasedoverhead.
NoteYoucanmixasmanyactiveandpassiveproxiesasyouwantinyourenvironment.Thisenablesyoutoexpandyourmonitoringsolutiontoreacheachpartofthenetworkandtohandlealargenumberofmonitoredobjects.Thisapproachkeepsthearchitecturesimpleandeasytomanagewithastrongcentralcoreandmanysimple,lightweightsatellites.
Ifyouwouldliketokeeptrackofalltheremainingitemsthattheproxyneedstosend,youcansetuptheproxytorunthisqueryagainstitsdatabase:
SELECT((SELECTMAX(proxy_history.id)FROMproxy_history)-nextid)FROMids
WHEREfield_name='history_lastid'
TipDownloadingtheexamplecode
Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
ThisquerywillreturnthenumberofitemsthattheproxystillneedstosendtotheZabbix
www.it-ebooks.info
server.ConsideringthatyouareusingMySQLasadatabase,youneedtoaddthefollowinguserparameterintheproxyagentconfigurationfile:
UserParameter=zabbix.proxy.items.sync.remaining,mysql-u<yourdbname
here>-p'<yourpasswordhere>'-e'SELECT((SELECTMAX(proxy_history.id)
FROMproxy_history)-nextid)FROMidsWHEREfield_name=history_lastid'2>&1
Now,allyouneedtodoissetanitemontheZabbixserversideandyoucanseehowyourproxyisfreeingitsqueue.
www.it-ebooks.info
InstallingZabbixZabbix,likealltheothersoftware,canbeinstalledintwoways:
1. Downloadthelatestsourcecodeandcompileit.2. Installitfrompackages.
Actually,thereisanotherwaytohaveaZabbixserverupandrunning:usingthevirtualappliance.TheZabbixserverappliancewillnotbeconsideredinthisbookasZabbixitselfdefinesthisvirtualapplianceasnotreadyforproductiveenvironments.Thisvirtualapplianceisnotaproductionreadysetupformanyreasons:
Itisamonolithwhereeverythingisinstalledonthesameserver.Thereisnoseparationfromthedatabaselayerandthepresentationlayer.Thismeansthateachoneofthesecomponentscanaffecttheperformanceoftheother.Thereisaclearwarningonthewebsitethatwarnsuswiththisstatement:TheApplianceisnotintendedforseriousproductionuseatthistime.
Ontheotherhand,theinstallationfrompackagesgivesussomebenefits:
ThepackagesmakeiteasytoupgradeandupdateDependenciesareautomaticallysortedout
Thesourcecodecompilationalsogivesussomebenefits:
WecancompileonlytheneededfeaturesWecanbuildtheagentstaticallyanddeployondifferentLinuxflavorsCompletecontrolonupdate
It’squiteusualtohavedifferentversionsofLinux,Unix,andMicrosoftWindowsonalargeenvironment.Thiskindofscenarioisquitediffusedonaheterogeneousinfrastructure,andifweusetheZabbix’sagentdistributionpackageoneachLinuxserver,wewillhavedifferentversionsoftheagentforsure,anddifferentlocationsfortheconfigurationfiles.
Themorethethingsarestandardizedacrossourserver,theeasieritwillbecometomaintainandupgradetheinfrastructure.The--enable-staticoptiongivesusawaytostandardizetheagentacrossdifferentLinuxversionsandrelease,whichisastrongbenefit.Theagent,staticallycompiled,canbeeasilydeployedeverywhereand,forsure,wewillhavethesamelocation(andwecanusethesameconfigurationfileapartfromthenodename)fortheagentandhis/herconfigurationfile.Theonlythingthatmightvaryisthestart/stopscriptandhowtoregisteritontherightinitrunlevel,butatleastthedeploymentwillbestandardized.
ThesamekindofconceptcanbeappliedtothecommercialUnix,bearinginmindtocompileitonthetargetenvironmentsothatthesameagentcanbedeployedondifferentUnixreleasesofthesamevendor.
www.it-ebooks.info
InstallingfrompackagesThefirstthingtodotoinstallZabbixfromrepoistoaddtheyumrepositorytoourlist.Thiscanbedonewiththefollowingcommand:
$rpm-Uvhhttp://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-
2.4-1.el6.noarch.rpm
Retrievinghttp://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-
2.4-1.el6.noarch.rpm
warning:/var/tmp/rpm-tmp.dsDB6k:HeaderV4DSA/SHA1Signature,keyID
79ea5ed4:NOKEY
Preparing…###########################################[100%]
1:zabbix-release###########################################[100%]
Oncethisisdone,wecantakeadvantageofallthebenefitsintroducedbythepackagemanagerandhavethedependenciesautomaticallyresolvedbyyum.
ToinstalltheZabbixserver,yousimplyneedtorun:
$yuminstallzabbix-server-mysqlzabbix-agentzabbix-javagateway
Now,youhaveyourserverreadytostart.Wecan’tstartitnowasweneedtosetupthedatabase,whichwillbedoneinthenextheading,anyway,whatyoucandoissetupthestart/stoprunlevelforourzabbix_serverandzabbix_agentdaemons:
$chkconfig--level345zabbix-serveron
$chkconfig--level345zabbix-agenton
Pleasedoublecheckifthepreviouscommandransuccessfullywiththefollowing:
$chkconfig--list|grepzabbix
zabbix-agent0:off1:off2:off3:on4:on5:on6:off
zabbix-server0:off1:off2:off3:on4:on5:on6:off
www.it-ebooks.info
SettingupaZabbixagentNow,asusuallyhappensinalargeserverfarm,itispossiblethatyouhavemanydifferentvariantsofLinux.Here,ifyoucan’tfindthepackageforyourdistribution,youcaneventhinktocompiletheagentfromscratch.Thefollowingarethestepsforthesame:
1. DownloadthesourcecodefromtheZabbixwebsite.2. Unpackthesoftware.3. Satisfyallthesoftwaredependencies,installingalltherelated-develpackages.4. Runthefollowingcommand:$./configure--enable-agent.
TipHere,youcanstaticallylinktheproducedbinarywiththe--enable-staticoption.Withthis,thebinaryproducedwillnotrequireanyexternallibrary.ThisisreallyusefultodistributetheagentacrossdifferentversionsofLinux.
Compileeverythingwith$make.
Now,beforeyourun$makeinstall,youcandecidetocreateyourownpackagetodistributewithCheckInstall.
www.it-ebooks.info
CreatingaZabbixagentpackagewithCheckInstallTheadviceistonotrunmakeinstall,butuseCheckInstalltoproducetherequiredpackageforyourLinuxOSfromhttp://asic-linux.com.mx/~izto/checkinstall/.
NoteWecanalsouseaprebuiltCheckInstall;thecurrentreleaseischeckinstall-1.6.2-20.2.i686.rpmonRedHat/CentOS.Thepackagewillalsoneedtherpm-buildpackage:
rpm-buildyuminstall
Also,weneedtocreatethenecessarydirectories:
mkdir-p~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
Thissoftwareenablesyoutocreateapackageformanydifferentversionsofthepackagemanager,namely,RPM,deb,andtgz.
NoteCheckInstallwillproducepackagesforDebian,Slackware,andRedHat,helpingustopreparetheZabbix’sagentpackage(staticallylinked)anddistributeitaroundourserver.
Now,weneedtoswitchtotherootaccountusing$sudosu–.Also,usethecheckinstallfollowedbytheseoptions:
$checkinstall--nodoc--install=yes-y
Ifyoudon’tfaceanyissue,youshouldgetthefollowingmessage:
******************************************************************
Done.Thenewpackagehasbeensavedto
/root/rpmbuild/RPMS/i386/zabbix-2.4.0-1.i386.rpm
Youcaninstallitinyoursystemanytimeusing:
rpm-izabbix-2*.4.0-1.i386.rpm
******************************************************************
Rememberthattheserverbinarieswillbeinstalledin<prefix>/sbin,utilitieswillbein<prefix>/bin,andthemainpagesunderthe<prefix>/sharelocation.
TipTospecifyadifferentlocationforZabbixbinaries,weneedtouse--prefixontheconfigureoptions(forexample,--prefix=/opt/zabbix).
www.it-ebooks.info
ServerconfigurationFortheserverconfiguration,weonlyhaveonefiletocheckandedit:
/etc/zabbix/zabbix_server.conf
Alltheconfigurationfilesarecontainedinthefollowingdirectory:
/etc/zabbix/
Allyouneedtochangefortheinitialsetupisthe/etc/zabbix/zabbix_server.confconfigurationfileandwritetheusername/passwordanddatabasenamehere.
NotePleasetakecaretoprotecttheaccesstotheconfigurationfilewithchmod400/etc/zabbix/zabbix_server.conf.
Thedefaultexternalscriptslocationis:
/usr/lib/zabbix/externalscripts
Also,thealertscriptdirectoryis:
/usr/lib/zabbix/alertscripts
Thiscanbechangedbyeditingthezabbix_server.conffile.
Theconfigurationontheagentsideisquiteeasy;basically,weneedtowritetheIPaddressofourZabbixserver.
www.it-ebooks.info
InstallingadatabaseThedatabasewewilluseonthisbook,asalreadyexplained,isMySQL.
Now,consideringthatyouhaveaRedHatserver,theproceduretoinstallMySQLfromtheRPMrepositoryisquiteeasy:
$yuminstallmysqlmysql-server
Now,youneedtosetuptheMySQLservicetostartautomaticallywhenthesystemboots:
$chkconfig--levels235mysqldon
$/etc/init.d/mysqldstart
TipRemembertosetapasswordfortheMySQLrootuser
Tosetapasswordfortheroot,youcanrunthesetwocommands:
/usr/bin/mysqladmin-urootpassword'new-password'
/usr/bin/mysqladmin-uroot-hhostname-of-your.zabbix.dbpassword'new-
password'
Alternatively,youcanrun:
/usr/bin/mysql_secure_installation
Thiswillalsohelpyoutoremovethetestdatabasesandanonymoususerdatathatwascreatedbydefault.Thisisstronglyrecommendedforproductionservers.
Now,it’stimetocreatetheZabbixdatabase.Forthis,wecanusethefollowingcommands:
$mysql-uroot-p
$mysql>CREATEDATABASEzabbixCHARACTERSETUTF8;
QueryOK,1rowaffected(0.00sec)
$mysql>GRANTALLPRIVILEGESonzabbix.*to'zabbixuser'@'localhost'
IDENTIFIEDBY'zabbixpassword';
QueryOK,0rowsaffected(0.00sec)
$mysql>FLUSHPRIVILEGES;
$mysql>quit
Next,weneedtorestorethedefaultZabbixMySQLdatabasefiles:
$mysql-uzabbixuser-pzabbixpasswordzabbix</usr/share/doc/zabbix-
server-mysql-2.4.0/create/schema.sql
$mysql-uzabbixuser-pzabbixpasswordzabbix</usr/share/doc/zabbix-
server-mysql-2.4.0/create/images.sql
$mysql-uzabbixuser-pzabbixpasswordzabbix</usr/share/doc/zabbix-
server-mysql-2.4.0/create/data.sql
Now,ourdatabaseisready.Beforewebegintoplaywiththedatabase,it’simportanttodosomeconsiderationaboutdatabasesizeandheavytasksagainstit.
www.it-ebooks.info
ConsideringthedatabasesizeZabbixusestwomaingroupsoftablestostoreitsdata:
HistoryTrends
Now,thespaceconsumedbythesetablesisinfluencedby:
Items:Thisisthenumberofitemsyou’regoingtoacquireRefreshrate:ThisisthemeanaveragerefreshrateofouritemsSpacetostorevalues:ThisdependsonRDBMS
Thespaceusedtostoredatacanvaryduetothedatabase,butwecanresumethespaceusedbythesetablesinthefollowingtable:
Typeofmeasure Retentionindays Spacerequired
History 30 10.8G
Events 1825(5years) 15.7GB
Trends 1825(5years) 26.7GB
Total NA 53.2GB
Thiscalculationis,ofcourse,doneconsideringtheenvironmentafter5yearsofretention.Anyway,weneedtohaveanenvironmentreadytosurvivethisperiodoftimeandretainthesameshapethatithadwhenitwasinstalled.Wecaneasilychangethehistoryandtrendsretentionpolicyperitem.Thismeansthatwecancreateatemplatewithitemsthathaveadifferenthistoryretentionbydefault.Normally,thehistoryissetto30days,butforsomekindofmeasure(suchasinwebscenarios)orotherparticularmeasures,weneedtokeepallthevaluesformorethanaweek.Thispermitsustochangethisvalueoneachitem.
www.it-ebooks.info
MySQLpartitioningNowthatweareawareofhowbigourdatabasewillbe,it’seasytoimaginethathousekeepingwillbeaheavytaskandthetime,CPU,andresourceconsumedbythisonewillgrowtogetherwiththedatabasesize.
Housekeepingisinchargetoremovetheoutdatedmetricsfromthedatabaseandtheinformationdeletedbyauser,andaswe’veseenthehistory,trends,andeventstablesare,aftersometime,hugetables.Thisexplainswhytheprocessissoheavytomanage.
Theonlywaywecanimproveperformancesoncewehavereachedthisvolumeofdataisbyusingpartitioninganddisablingthehousekeeperaltogether.
Partitioningthehistoryandtrendtableswillprovideuswithmanymajorbenefits:
Allhistorydatainatableforaparticulardefinedwindowtimeareself-containedinitsownpartition.Thisallowsyoutoeasilydeleteolddatawithoutimpactingthedatabaseperformance.WhenyouuseMySQLwithInnoDB,andifyoudeletedatacontainedinatable,thespaceisnotreleased.Thespacefreedismarkedasfree,butthediskspaceconsumedwillnotchange.Whenyouusepartition,andifyoudropapartition,thespaceisimmediatelyfreed.Queryperformancecanbeimproveddramaticallyinsomesituations,inparticular,whenthereisheavyaccesstothetable’srowsinasinglepartition.Whenaqueryupdatesahugeamountofdataorneedsaccesstoalargepercentageofthepartition,thesequentialscanisoftenmoreefficientthantheindexusagewitharandomaccessorscatteredreadsagainstthisindex.
Unfortunately,Zabbixisnotabletomanagethepartitions.So,weneedtodisablehousekeeping,anduseanexternalprocesstoaccomplishhousekeeping.
Whatweneedtohaveisastoredprocedurethatdoesalltheworkforus.
Thefollowingisthestoredprocedure:
DELIMITER$$
CREATEPROCEDURE`partition_maintenance`(SCHEMA_NAMEVARCHAR(32),
TABLE_NAMEVARCHAR(32),KEEP_DATA_DAYSINT,HOURLY_INTERVALINT,
CREATE_NEXT_INTERVALSINT)
BEGIN
DECLAREOLDER_THAN_PARTITION_DATEVARCHAR(16);
DECLAREPARTITION_NAMEVARCHAR(16);
DECLARELESS_THAN_TIMESTAMPINT;
DECLARECUR_TIMEINT;
Untilhere,wehavedeclaredthevariableweneedafter.Now,onthenextline,wewillcallthestoredprocedureresponsibletocheckwhetherapartitionisalreadypresentandifnot,wewillcreatethem:
CALLpartition_verify(SCHEMA_NAME,TABLE_NAME,HOURLY_INTERVAL);
SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(NOW(),'%Y-%m-%d
00:00:00'));
www.it-ebooks.info
IFDATE(NOW())='2014-04-01'THEN
SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD(NOW(),
INTERVAL1DAY),'%Y-%m-%d00:00:00'));
ENDIF;
SET@__interval=1;
create_loop:LOOP
IF@__interval>CREATE_NEXT_INTERVALSTHEN
LEAVEcreate_loop;
ENDIF;
SETLESS_THAN_TIMESTAMP=CUR_TIME+(HOURLY_INTERVAL*@__interval
*3600);
SETPARTITION_NAME=FROM_UNIXTIME(CUR_TIME+HOURLY_INTERVAL*
(@__interval-1)*3600,'p%Y%m%d%H00');
Nowthatwehavecalculatedalltheparametersneededbythecreate_partitionprocedure,wecanrunit.Thisstoredprocedurewillcreatethenewpartitiononthedefinedschema:
CALLpartition_create(SCHEMA_NAME,TABLE_NAME,
PARTITION_NAME,LESS_THAN_TIMESTAMP);
SET@__interval=@__interval+1;
ENDLOOP;
SETOLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(),INTERVAL
KEEP_DATA_DAYSDAY),'%Y%m%d0000');
Thesectionthatfollowsisresponsibletoremovetheolderpartitions,usingtheOLDER_TAN_PARTITION_DATEprocedure,whichwehavecalculatedonthelinesbefore:
CALLpartition_drop(SCHEMA_NAME,TABLE_NAME,
OLDER_THAN_PARTITION_DATE);
END$$
DELIMITER;
Thisstoredprocedurewillbethecoreofourhousekeeping.Itwillbecalledwiththefollowingsyntax:
CALLpartition_maintenance('<zabbix_db_name>','<table_name>',
<days_to_keep_data>,<hourly_interval>,<num_future_intervals_to_create>)
Theprocedureworksbasedon1hourintervals.Next,ifyouwanttopartitiononadailybasis,theintervalwillbe24hours.Instead,ifyouwant1hourpartitioning,theintervalwillbe1.
Youneedtospecifythenumberofintervalsthatyouwantcreatedinadvance.Forexample,ifyouwant2weeksintervaloffuturepartitions,use14.Ifyourintervalis1(forhourlypartitioning),thenthenumberofintervalstocreateis336(24*14).
Thisstoredprocedureusessomeotherstoresprocedures:
partition_create:Thiscreatesthepartitionforthespecifiedtablepartition_verify:Thischeckswhetherthepartitionisenabledonatable,ifnot,thencreateasinglepartitionpartition_drop:Thisdropspartitionsolderthanatimestamp
Forallthedetailsaboutthesestoredprocedures,seeAppendixA,PartitioningtheZabbix
www.it-ebooks.info
Database.
Onceyou’vecreatedalltherequiredstoredprocedures,youneedtochangetwoindexestoenabletheminordertobereadyforapartitionedtable:
mysql>Altertablehistory_textdropprimarykey,addindex(id),drop
indexhistory_text_2,addindexhistory_text_2(itemid,id);
QueryOK,0rowsaffected(0.49sec)
Records:0Duplicates:0Warnings:0
mysql>Altertablehistory_logdropprimarykey,addindex(id),dropindex
history_log_2,addindexhistory_log_2(itemid,id);
QueryOK,0rowsaffected(2.71sec)
Records:0Duplicates:0Warnings:0
Oncethisisdone,youneedtoschedulethepartition_maintenance_allstoredprocedurewithacronjob.Formoredetailsaboutthepartition_maintenance_allprocedure,pleasechecktheinstructionscontainedinAppendixA,PartitioningtheZabbixDatabase.Thecronjobneedstoexecutethefollowingcommand:
mysql-h<zabbix_db_host>-u<zabbixuser>-p<zabbixpassword>zabbixdatabase
-e"CALLpartition_maintenance_all('zabbix');"
Oncethishasbeenset,youneedtobearinmindtodisablethehousekeepingforhistoryandtrends.VerifythattheOverrideitem<trend/history>periodZabbixconfigurationischeckedforbothhistoryandtrends.Here,youneedtosettheDatastorageperiod(indays)boxforhistoryandtrendstothevalueyou’vedefinedinyourprocedure,ourexampleinAppendixA,PartitioningtheZabbixDatabaseisof28and730.
www.it-ebooks.info
InstallingaZabbixproxyInstallationoftheZabbixproxyfrompackagesisaquitesimpletask.Onceyou’veaddedtheZabbixrepository,youonlyneedtorunthefollowingcommand:
$yuminstallzabbix-proxy-mysql
Thiswillinstalltherequiredpackages:
Installation:
zabbix-proxy-mysqlx86_642.4.0-1.el6zabbix390k
Installingfordependencies:
zabbix-proxyx86_642.4.0-1.el6zabbix21k
TheZabbixproxyinstallationisquitesimilartotheserverone.Onceyou’veinstalledtheserver,youneedtoinstallMySQL,createthedatabase,andimporttheDBschema:
$mysql-uroot-p
$mysql>CREATEDATABASEzabbixCHARACTERSETUTF8;
QueryOK,1rowaffected(0.00sec)
$mysql>GRANTALLPRIVILEGESonzabbix.*to'zabbixuser'@'localhost'
IDENTIFIEDBY'zabbixpassword';
QueryOK,0rowsaffected(0.00sec)
$mysql>FLUSHPRIVILEGES;
$mysql>quit
Next,weneedtorestorethedefaultZabbixMySQLdatabasefiles:
$mysql-uzabbixuser-pzabbixpasswordzabbix</usr/share/doc/zabbix-
proxy-mysql-2.4.0/create/schema.sql
Now,weneedtostartthedatabase,configuretheproxy,andstarttheservice.Inthisexample,wehaveconsideredtouseaZabbixproxythatreliesonaMySQLwithInnoDBdatabase.Thisproxycanbeperformedintwodifferentways:
Lightweight(andthenuseSQLite3)Robustandsolid(andthenuseMySQL)
Here,wehavechosenthesecondoption.Inalargenetworkenvironmentwheretheproxy,incaseofissue,needstopreserveallthemetricsacquireduntiltheserveracquiresthemetrics,it’sbettertoreduce,attheminimum,theriskofdataloss.Also,ifyouconsiderthisscenarioinalargenetworkenvironment,youmostlikelywillhavethousandsofsubnetworksconnectedtotheZabbixserverwithallthepossiblenetworkdevicesin-between.Well,exactly,thisisnecessarytouseadatabasethatcanpreventanydatacorruptions.
www.it-ebooks.info
InstallingtheWebGUIinterfaceTheWebGUIinterfacewillbedoneoncemoreusingtheRPMs.
Toinstallthewebinterface,youneedtorunthefollowingcommand:
$yuminstallzabbix-web-mysql
Yumwilltakecaretoresolveallthedependencies.Onceyou’redone,theprocessofthiscomponentisquiteeasy:weneedtoopenawebbrowser,pointatthefollowingURL:http://your-web-server/zabbix,andfollowtheinstructions.
OnthestandardRedHatsystem,yousimplyneedtochangetheseparametersonyour/etc/php.inifile:
php_valuemax_execution_time300
php_valuememory_limit128M
php_valuepost_max_size16M
php_valueupload_max_filesize2M
php_valuemax_input_time300
Also,setyourtimezoneonthesamefile(forexample,php_valuedate.timezoneEurope/Rome).
Now,it’stimetostartupApache,butbeforethis,weneedtocheckwhetherwehaveSELinuxenabledandonwhichmode?TocheckyourSELinuxstatus,youcanrun:
#sestatus
SELinuxstatus:enabled
SELinuxfsmount:/selinux
Currentmode:permissive
Modefromconfigfile:permissive
Policyversion:24
Policyfromconfigfile:targeted
Now,youneedtocheckwhetheryouhavethehttpddaemonenabledtousethenetworkwiththefollowingcommand:
#getseboolhttpd_can_network_connect
httpd_can_network_connect-->off
Mostlikely,youwillhavethesamekindofresult,thenallweneedtodoisenablethehttpd_can_network_connectoptionusingthenextcommandwith–Ptopreservethevalueafterareboot:
#setsebool–Phttpd_can_network_connecton
#getseboolhttpd_can_network_connect
httpd_can_network_connect-->on
Now,allthatwestillhavetodoisenablethehttpddaemonandstartourhttpdserver:
#servicehttpdstart
Startinghttpd:[OK]
Next,enablethehttpdserverasaservice:
www.it-ebooks.info
#chkconfighttpdon
Wecancheckthechangedonewiththenextcommand:
#chkconfig--listhttpd
httpd0:off1:off2:on3:on4:on5:on6:off
Onceyou’vedonethis,youonlyneedtofollowthewizard,andinafewclicks,youwillhaveyourwebinterfacereadytostartup.
TipIfyouknowthattheloadagainstthewebserverwillbehigh,duetoahighnumberofaccountsthatwillaccessit,probably,it’sbettertoconsiderusingNginx.
Now,youcanfinallystartyourZabbixserverandthefirstentryinthe/var/log/zabbix/zabbix_server.logfilewilllooksomethinglikethefollowingcode:
37909:20140925:091128.868StartingZabbixServer.Zabbix2.4.0(revision
48953).
37909:20140925:091128.868******Enabledfeatures******
37909:20140925:091128.868SNMPmonitoring:YES
37909:20140925:091128.868IPMImonitoring:YES
37909:20140925:091128.868WEBmonitoring:YES
37909:20140925:091128.868VMwaremonitoring:YES
37909:20140925:091128.868Jabbernotifications:YES
37909:20140925:091128.868EzTextingnotifications:YES
37909:20140925:091128.868ODBC:YES
37909:20140925:091128.868SSH2support:YES
37909:20140925:091128.868IPv6support:YES
37909:20140925:091128.868******************************
37909:20140925:091128.868usingconfigurationfile:
/etc/zabbix/zabbix_server.conf******************************
Next,youcanstarttoimplementandacquirealltheitemscriticalforyournetwork.
www.it-ebooks.info
SummaryInthischapter,wecoveredalargenumberofcomponents.Westartedwithdefiningwhatalargeenvironmentis.Wealsosawhowthenetworksetupcanbedesignedandhowitcanevolvewithinyourinfrastructure.Wesawtheheaviesttaskontheserverside(housekeeping)andhowtoavoidperformancedegradationduetothis.WediscussedMySQLpartitioningin-depth.Wealsobrieflydiscussedthedifferencesbetweenactiveandpassiveproxies;youwillnowbeabletodecidehowtosetthemupandwhichonetochooseonceyouknowyournetworktopology.Also,wesawhowtoacquiresomecriticalmetricstomonitortheZabbixproxyconnectionandtheamountofitemsthatitstillneedstosendus.
Asyoucansee,wecoveredalotofargumentsinjustonechapter;wedidthisbecausewewouldliketousemorespaceintheupcomingchapters.Inthenextchapter,wewillexplorethedifferentappliancesandprotocolsatlayer2andlayer3oftheISO/OSIstack.Also,youwillseehowtobestextrapolatemeaningfulmonitoringdatafromthecollectedmeasurefortheprotocollayers2and3.
www.it-ebooks.info
Chapter2.ActiveMonitoringofYourDevicesNowthatyouhaveaworkingZabbixsetup,it’stimetotakealookatyournetworkandfigureoutthecomponentsthatyouwanttomonitor,thekindofdatayouwanttocollect,andtheconditionsunderwhichyouwanttobenotifiedaboutproblemsandstatechanges.
Itwouldbeimpossibleforanybookonthistopictofullycoverallthedifferentkindsofnetworkappliancesandtopologiesandallthedifferentmonitoringscenariosthatanetworkadministratormightneedaseveryenvironmenthasitsownspecificquirksthatagoodmonitoringsolutionhastoaccountfor.ThischapterwillofferyouafewexamplesofthedifferentmonitoringpossibilitiesZabbixcanachievebyrelyingondifferentmethodsandprotocols.You’llseehowtoqueryyournetworkfromthedatalinklayeruptoroutingandnetworkflowusingICMP,SNMP,andlogparsingfacilitiestocollectyourmeasurements.
You’lllearnhowtoextractmeaningfulinformationfromthedatayougatheredusingaggregatedandcalculateditemsandhowtoconfigurecomplextriggersthatwillalertyouaboutrealnetworkissueswhileminimizinguninterestingornonrelevantdata.
Bytheendofthechapter,you’llhaveagoodoverviewofZabbix’snetworkmonitoringpossibilities,andyou’llbereadytoadaptwhatyoulearnedforyourspecificrequirements.Butlet’sfirsthaveaquickoverviewofhowZabbixorganizesmonitoringdatawithhosts,templates,items,andtriggers.
www.it-ebooks.info
UnderstandingZabbixhostsOneofZabbix’sgreatstrengthsisitsflexibilitywhenitcomestoorganizingmonitoringdata.Evenwithoutconsideringitspowerfultemplatinganddiscoveryfeatures,whichwillbecoveredinChapter4,DiscoveringYourNetwork,thereisalotthatyoucandowithstandardhosts,items,andtriggers.Hereareafewtipsonhowyoucanusethemeffectively.
www.it-ebooks.info
HostsandhostgroupsZabbixhostsusuallyrepresentasingle,specificboxorapplianceinyournetwork.Theycanalsobeapartofoneormorehostgroups.
HostgroupsareveryusefulastheymakeiteasytonavigateZabbix’sinterface,separatinghostsintocategoriesandallowingyoutoorganizeandmanageahugeamountofapplianceswithouthavingtodealwithimpossiblylonglistsofhostnames.Thesamehostcanbepartofdifferenthostgroups,andthiscanbeveryusefulasyoumightwant,forexample,tohaveagroupforallyourrouters,agroupforallyourswitches,andagroupforeverysubnetyoumanage.So,asinglerouterwillbepartoftheroutersgroupandallthesubnetgroupsithasaninterfaceon,whileaswitchwillbepartoftheswitchesgroupandofthesubnetit’spartof,andsoon.
Whilethisiscertainlyagoodwaytoorganizeyourhosts,bothtovisualizeandtomanageyourmonitoringdata,thereareacoupleofnot-too-obviouspitfallsyoushouldbeawareofifyoudecidetoputthesamehostinmultiplegroups:
Calculateditemsshowaggregatemonitoringdatabasedonhostgroupmembership.Ifyouconfigureanaggregateditemthatusesmorethanonecalculateditemfromdifferenthostgroups,youcanendupusingthesamehost’sdatamorethanonce,introducingasignificanterrorinyourcalculations.Actionsareusuallyfilteredbasedonhostgroups.Thismeansthatthesametriggereventcouldfireupmorethanoneactionifthehostispartofmorethanonehostgroup,leadingtopotentiallyduplicatemessagesandalerts.Useraccesspermissionsarehost-group-based.Thismeansthatsomeuserscouldbeabletoseemorehostsandmonitoringdatathantheyactuallyneedtoifahostendsupinahostgrouptheyhaveaccessto.
Thisisbynomeansanattempttodiscouragethepracticeofassigningmultiplehostgroupstothesamehost.Justbeawareoftheramificationsofsuchapracticeanddon’tforgettotakeintoconsiderationtheaddedcomplexitywhenyouconfigureyouritems,actions,andaccesspermissions.
HostinterfacesEachhostiscomposedofacollectionofitemsthatrepresenttherawmonitoringdata,andtriggers,whichrepresentZabbix’smonitoringintelligencebasedonthedatagathered.It’salsocomposedofaseriesofinterfacesthattelltheZabbixserverorproxyhowtocontactthehosttocollecttheaforesaidmonitoringdata.Mostnetworkapplianceshavemorethanoneinterface,soyouwouldwanttomakesurethatallhoststhatrepresentrouters,firewalls,proxies,gateways,andwhatnot,arelistingallthoseappliances’interfacesandtheiraddresses.Theadvantagesareobvious:
You’llbeabletoquicklyreviewwhataddressesareconfiguredonaspecifichostwhilelookingatmonitoringdataYou’llbeabletodifferentiateyourchecksbyqueryingdifferentaddressesorportsofthesamehostbasedonyourneeds
www.it-ebooks.info
Yourmapsandtopologieswillbemoreconsistentwithwhat’sactuallydeployed
Addinginterfacestoahostisfairlystraightforward.AllyouneedtodoisnavigatetoConfiguration|Hostsandthenselectthehostyouwanttoedit.Theinterfacessectionisinthemainconfigurationtab,asshowninthefollowingscreenshot:
Asyoucanseeintheaboveexample,therearethreeagentinterfacesthatshowallthenetworkstherouterisconnectedtoandjustoneSNMPinterface.AgentinterfacesareusednotonlyforZabbixagentitems,butalsoforsimpleandexternalchecks.Ontheotherhand,you’lluseSNMPinterfacestosendSNMPqueriestoyourhost.Theprecedingexampleassumesthatyou’llonlyuseSNMPontherouter’sinterfacethatisconnectedtoamanagementnetwork(192.168.1.0inthisexample),whileyou’llalsouseICMP,TCP,andexternalchecksonitstwoproductioninterfaces.Ofcourse,youarefreetoconfiguredifferentIPaddressesforAgentandSNMPinterfacesdependingonwhatprotocolsandchecksyouplantoactivateonwhichinterfaces.
HostinventoryHavinginventorydatadirectlyavailableinyourmonitoringsolutionhasalotofobviousadvantageswhenitcomestoattachingusefulinformationtoyouralertsandalarms.Unfortunately,themorehostsyouhavetomanage,themoreessentialitistohaveup-to-dateinventoryinformation,andtheharderitistomaintaintheaforesaidinformationinareliableandtimelymanner.Manuallyupdatingahost’sinventorydatacanquicklybecomeanimpossibletaskwhenyouhavetensorhundredsofhoststomanage,andit’snotalwayspossibletowriteautomatedscriptsthatwilldothejobforyou.Fortunately,Zabbixoffers
www.it-ebooks.info
anautomaticinventoryfeaturethatcanatleastpartiallyfillininventorydatabasedonactualmonitoringdata.Toactivatethisfeature,firstyou’llneedtoselectAutomaticintheHostinventorytabofahostconfigurationpageandthenmovetotheitemsthatyou’llusetopopulatetheinventorydata.
Whenconfiguringanitem,youshouldassignitsdatatoaspecificinventoryfieldsothattheaforesaidfield’svaluewillbesetandautomaticallyupdatedbasedontheitem’smeasurements,asshowninthefollowingscreenshot:
Asyoucanseeintheprecedingexample,ahost’slocationinventoryvaluewillbepopulatedbasedonthecorrespondingSNMPquery.Thismeansthatifyouchangeadevice’slocationinformation,thatchangewillbereflectedinZabbixassoonastheitem’svalueispolledonthedevice.Dependingonthedataavailableonthedevice,you’llbeabletopopulateonlyafewinventoryfieldsormostofthem,whilefallingbackonmanualupdatesofthefieldsthatfalloutsideofyourdevice’sreportingpossibilities.
Speakingofitems,let’snowfocusonthedifferentmonitoringpossibilitiesthatZabbixitemsofferandhowtoapplythemtoyourenvironment.
www.it-ebooks.info
GoingbeyondZabbixagentsTherearecertainlymanyadvantagesinusingZabbix’sownagentsandprotocolwhenitcomestomonitoringWindowsandUnixoperatingsystemsortheapplicationsthatrunonthem.However,whenitcomestonetworkmonitoring,thevastmajorityofmonitoredobjectsarenetworkappliancesofvariouskinds,whereit’softenimpossibletoinstallandrunadedicatedagentofanytype.Thisbynomeansimpliesthatyou’llbeunabletofullyleverageZabbix’spowertomonitoryournetwork.Whetherit’sasimpleICMPechorequest,anSNMPquery,anSNMPtrap,netflowlogging,oracustomscript,therearemanypossibilitiestoextractmeaningfuldatafromyournetwork.Thissectionwillshowyouhowtosetupthesedifferentmethodsofgatheringdata,andgiveyouafewexamplesonhowtousethem.
www.it-ebooks.info
SimplechecksLet’sstartwiththesimplestcase.Atfirstglance,simplechecksdon’tlookthatinteresting:excludingalltheVMwareHypervisorchecksthatareincludedinthiscategory,simplechecksarereducedtoacoupleofgenericTCP/IPconnectionchecksandthreeICMPechochecks,asfollows:
Checkname Description
Icmpping Thisreturns1ifthehostrespondstoanICMPping;0otherwise
Icmppingloss ThisreturnsthepercentageoflostICMPpingpackets
Icmppingsec ThisreturnstheICMPresponsetimeinseconds
Net.tcp.service Thisreturns1ifthehostacceptsconnectionsonaspecifiedTCPport;0otherwise
Net.tcp.service.perf ThisreturnsthenumberofsecondsspenttoobtainaconnectiononaspecifiedTCPport
Generallyspeaking,thesechecksprovemoreusefulasthedistancebetweenthemonitoringprobeandthemonitoredhostincreases,bothintermsofphysicaldistance(ageographicallinktoanothercityforexample)andintermsofhopsapackethastogothrough.Thismeansthatifyouareinterestedinyournetwork’sperformance,itwouldmakesensetoassignhostswithsimplecheckstoZabbixproxiesthatarenotinthesamesubnet,butaresituatedwheretheywillmimicascloselyaspossibleyouractualnetworktraffic.Net.tcp.serviceisparticularlyusefulfromthispointofview,notjusttocheckthestatusoftheavailabilityofspecificserviceswhenyoucannotuseZabbixagents,butalsotocheckgeneralhostavailabilityacrossrestrictivefirewallsthatblockICMPtraffic.
TipInordertoreducenetworktrafficandtomakemoreefficientICMPchecks,Zabbixusesfpinginsteadoftheregularpingwhenexecutingicmpping,icmppingloss,andicmppingsecitemchecks.
MakesureyouhavefpinginstalledonyourZabbixserverandalsoonalltheZabbixproxiesthatmightneedit.Ifyoudon’thaveit,asimpleyuminstallfpingwillusuallybeenoughfortheZabbixdaemonstofinditanduseit.
Whilebothnet.tcp.serviceandnet.tcp.service.perfdosupportsomewell-knownprotocols,suchasSSH,FTP,HTTP,andsoon,thesetwoitems’mostusefuloptionisprobablytheonethatallowsyoutoperformasimpleTCPhandshakeconnectionandcheckwhetheraspecificIPisreachableonaspecificport.Thesekindofchecksareusefulbecause,justlikeICMPpings,theywillmostlyinvolvethenetworkstack,reducingapplicationoverheadtoaminimum,thusgivingyoudatathatmorecloselymatchesyouractualnetworkperformance.Ontheotherhand,unlikeICMPpings,theywillallowyoutocheckforTCPportavailabilityforagivenhost.Obvioususecasesincludemakinglightweightservicechecksthatwillnotimpactverybusyhostsorappliancestoomuch,
www.it-ebooks.info
andmakingsurethatagivenfirewallisallowingtrafficthrough.
Aslightlylessobvioususecaseisusingoneormorenet.tcp.serviceitemstomakesurethatsomeservicesarenotrunningonagiveninterface.Takeforexample,thecaseofaborderrouterorfirewall.Unlessyouhavesomeveryspecialandspecificneeds,you’lltypicallywanttomakesurethatnoadminconsolesareavailableontheexternalinterfaces.Youmighthavedouble-checkedtheappliance’sinitialconfiguration,butasystemupdate,acarelessadmin,orasecuritybugmightchangetheaforesaidconfigurationandopenyourappliance’sadmininterfacestoafarwideraudiencethanintended.AsecuritybreachlikethisonecouldpassunobservedforalongtimeunlessyouconfigureafewsimpleTCP/IPchecksonyourappliance’sexternalinterfacesandthensetupsometriggersthatwillreportaproblemifthosechecksreportanopenandresponsiveport.
Let’staketheexampleoftherouterwithtwoproductioninterfacesandamanagementinterfaceshowninthesectionabouthostinterfaces.Iftherouter’sHTTPSadminconsoleisavailableonTCPport8000,you’llwanttoconfigureasimplecheckitemforeveryinterface:
Itemname Itemkey
management_https_console net.tcp.service[https,192.168.1.254,8000]
zoneA_https_console net.tcp.service[https,10.10.1.254,8000]
zoneB_https_console net.tcp.service[https,172.16.7.254,8000]
Allthesecheckswillreturn1iftheserviceisavailable,and0iftheserviceisnotavailable.Whatchangesishowyouimplementthetriggersontheseitems.Forthemanagementitem,you’llhaveaproblemiftheserviceisnotavailable,whilefortheothertwo,you’llhaveaproblemiftheserviceisindeedavailable,asshowninthefollowingtable:
Triggername Triggerexpression
Managementconsoledown {it-1759-r1:net.tcp.service[http,192.168.1.254,8000].last()}=0
ConsoleavailablefromzoneA {it-1759-r1:net.tcp.service[http,10.10.1.254,8000].last()}=1
ConsoleavailablefromzoneB {it-1759-r1:net.tcp.service[http,172.16.7.254,8000].last()}=1
Thisway,you’llalwaysbeabletomakesurethatyourdevice’sconfigurationwhenitcomestoopenorclosedportswillalwaysmatchyourexpectedsetupandbenotifiedwhenitdivergesfromthestandardyouset.
Tosummarize,simplechecksaregreatforallcaseswhereyoudon’tneedcomplexmonitoringdatafromyournetworkastheyarequitefastandlightweight.Forthesamereason,theycouldbethepreferredsolutionifyouhavetomonitoravailabilityforhundredstothousandsofhostsastheywillimpartarelativelylowoverheadonyour
www.it-ebooks.info
overallnetworktraffic.
Whenyoudoneedmorestructureandmoredetailinyourmonitoringdata,it’stimetomovetothebreadandbutterofallnetworkmonitoringsolutions:SNMP.
www.it-ebooks.info
KeepingSNMPsimpleTheSimpleNetworkMonitoringProtocol(SNMP)isanexcellent,generalpurposeprotocolthathasbecomewidelyusedbeyonditsoriginalpurpose.Whenitcomestonetworkmonitoringthough,it’salsooftentheonlyprotocolsupportedbymanyappliances,soit’softenaforced,albeitnaturalandsensible,choicetointegrateitintoyourmonitoringscenarios.Asanetworkadministrator,youprobablyalreadyknowallthereistoknowaboutSNMPandhowitworks,solet’sfocusonhowit’sintegratedintoZabbixandwhatyoucandowithit.
Firstofall,we’llneedtotalkaboutSNMPgetsandSNMPtrapsintwodifferentdiscussionsastheyareimplementedandusedindifferentwaysbyZabbix.ThereasonforthisseparationisintheverynatureofSNMPgetsasopposedtoSNMPtraps.AnSNMPgetrepresentsasingle,discretepieceofinformationthatrepresentsthecurrentstatusofametric,andit’snottiedtoanyspecificevent.Whetherit’sacounterwiththetotalnumberofbytesthatpassedthroughaninterface,aBooleanvaluethatwilltellifalinkisupordown,orastringwithanappliance’slocationorcontactinformation,anSNMPvaluewillbeavailableatanymoment,anditwillbepossibletopollitwithanarbitraryfrequency.
ThismapsnicelytoZabbixitems.JustlikeSNMPgetvalues,theyalsorepresentsingle,discretevaluesthatcanbepolledwitharbitraryfrequency.ThismakesitreallystraightforwardtouseregularSNMPqueriestopopulateZabbixitemssincetheonlythingsyouhavetoworryaboutaretheSNMPOID,thedatatype,andthecommunitystringorauthenticationinformation.We’llseeafewexamplesinthenextparagraph.
AnSNMPtraprepresentsaspecificeventthathappensataspecificpointintime.Itmightrepresentalinkstatechange,arebootevent,orauserlogin.Inanycase,youcannotquerythestateofanSNMPtrap;youjusthavetowaittoreceiveone,anditwillnotrepresentasingle,discretevaluebutachangefromonevaluetoanother.Theyresemble,inmanyways,Zabbixeventsinsteadofrawdata.ThiscomplicatesthingsalittlesinceZabbixeventsaretheresultofevaluatingtriggersagainstcollecteddata,whileSNMPtrapscanonlyenterZabbixasitemvalues,thatis,ascollecteddata.Sowe’llneedtoresolvethisapparentmismatchinordertofullyleveragetheinformationcontainedinSNMPtraps.We’llseehowinashortwhile,butfirstlet’slookatafewdetailsconcerningregularSNMPqueriesexecutedfromZabbix.
GettingSNMPdataintoZabbixAZabbixserverusuallycomeswithgoodSNMPsupportoutofthebox.Notonlydoesitsupportthequeryingprotocolnatively,butitalsocomesequippedwithanumberofSNMPtemplatesthatcangetyoustartedintherightdirection.ThismeansthatformostdevicesyouonlyhavetolinktheTemplateSNMPDevicetemplate,andyou’llimmediatelybeabletogetsomebasicinformationaboutit,asshowninthefollowingscreenshot:
www.it-ebooks.info
We’vealreadyseenhowtheDevicelocationitemcanbeusedtopopulateahost’sinventorylocationrecord,butthereareacoupleofotherusefulbitsofinformationintheabovepicture.
Firstofall,there’salow-leveldiscoveryruletoexplore.We’lldelvemoredeeplyintodiscoveryrulesinChapter4,DiscoveringYourNetwork,butfornow,we’lljustseethatit’saboutdynamicallycreatingnetworkinterfaceitems:
Foreveryinterface,eightitemswillbecreated,includingtheinterfacename,operationalstatus,incomingandoutgoingtraffic,andsoon.Thismeansthatthesametemplatewillbeusefulforthebasicmonitoringofnetworkapplianceswithanynumberofnetworkinterfaces.
Thesecondthingtonotice,lookingatbothimages,istheupdateinterval,andhistoryandtrendretentionperiodsfortheitems.Zabbixtriestosetsomesensibledefaults,butyou’llprobablyneedtoupdatesomeofthosevaluesbasedonthenumberofmonitoredhostsyouhaveinyourenvironment,yourstoragespaceavailability,andthenetworkloadofyourmonitoringtraffic.
NoteAnotherparameterthatisrelatedtoZabbix’sperformanceistheinitial(andminimum)numberofpollersthattheserverkeepsactiveatanygiventime.Ifyoufindthatyourpollingqueueisgettinglonger,youmightwanttoincreasethenumberofpollersinzabbix_server.conf.Theavailabledefaultoptionsare:
www.it-ebooks.info
#StartPollers=5
#StartIPMIPollers=0
#StartPollersUnreachable=1
#StartTrappers=5
#StartPingers=1
#StartDiscoverers=1
#StartHTTPPollers=1
Workyourwayupslowly,oryou’lljustendupwithunnecessaryprocessesbeingcreatedwhenZabbixisstarted.
Ifyouhavehundredsofhoststomonitor,andforeveryhost,youcollecttensofsinglemeasurementseveryminute,youwouldreachapointwhereyourZabbixserver’snetworkloadorCPUloadwillstarttoimpactontheserver’sperformance,leadingtodelaysinitempollingordroppedconnections.Ifyoucannotjustupgradetomorepowerfulhardware,youmighthavetotweakthepollingintervalofyourtemplatessothattheystrikeagoodbalancebetweengranularityofdetailandperformance.
Adevice’sname,contactdetails,description,location,andsuchlike,willrarelychangeoncethedevicehasbeendeployed,soitwouldbeawastetopollforthosevalueseveryhour(3,600seconds).Bychangingtheintervalto6hoursorevenaday,you’llautomaticallyreduceyournetworktrafficrelatedtoessentiallyfixedinformationbyafactorof6,upto24.
Raisingthepollingintervalforsomeoftheinterfacecounterscanhaveanevenmoredramaticimpactonyoursystemandnetworkload.Whileyou’llprobablywanttochecktheadminandoperationalstatusofaninterfaceasoftenaspossible—otherwiseyouruntheriskofnotgettingnotifiedaboutpossibleproblemsinatimelymanner—ontheotherhand,you’llprobablybeabletolivewithpollingincomingandoutgoingtrafficanderrorseveryfiveminutes(300seconds)insteadofeveryminute.Yourgraphswillstillbeverydetailed,butyournetworkwillbemuchlessfloodedwithSNMPrequests.Keepinmindthatchangeslikethesemightnotseemmuchwhenreferredtoasinglehost,butasthenumberofyourmonitoredobjectsgrow,youcanveryquicklyrunuptohundredsoreventhousandsofnewmonitoringvaluespersecondcomingintoyourZabbixserver.
Thesamecanbesaidwhenitcomestoretentionperiodsandstoragespace.Inthiscase,keepinmindthattrendsstoreaboutthreevaluesperhour(min,maxandaverage)overthetimerangespecified,whilehistorystoresallvaluescollectedinthespecifiedtimerange.Thismeansthatbasedonyourpollinginterval,it’susuallycheapertoextendatrendretentionvaluethanahistoryone.Thisis,ofcourse,validonlyfornumericalvaluesasstringonescan’treallyhavetrends,justhistory.
OnelastthingtonoticeintheaboveimagesisthatthemonitoringprotocolforallitemsissettoSNMPv2.JustlikeSNMPv1,SNMPv2doesn’tofferrealsecurityforthemonitoringdatathatcrossesthenetworkbetweenanapplianceandthemonitoringserver:alltrafficissentandreceivedintheclear,andtheSNMPcommunityisjustastring,easilyparsablefrominterceptedtraffic.Whileit’scertainlytruethatafewnetworkappliancesdon’tsupportSNMPv3becauseeithertheyaretoooldortheyaretoosimple,It’salsotruethat
www.it-ebooks.info
thenewversionoftheprotocolhasbeenaroundforquiteawhilenowandanumberofappliancesdosupportit.ThemainadvantagesofSNMPv3areitsauthenticationandencryptioncapabilities.Thesecanhelpmakesurethatallmonitoringtrafficisnotbogusorcorrupted,andthatit’skeptconfidentialfrompryingeyes.Thisisparticularlyimportantifyouneedtomonitorsomehostsoveranetworklinkyouhavenorealcontrolover,suchasaWANconnectionthroughathird-partyprovider.ItwouldalwaysbenicetouseSNMPv3acrossyournetwork,butincaseslikethese,youarestronglyencouragedtodosoasthere’sarealpossibilitythatyourtrafficcanbeindeedinterceptedandtappedinto.
Let’staketheexampleofaCiscorouter,andlet’sseehowtoconfigureSNMPv3onitbeforemovingontotheZabbixside.
Firstofall,let’screateamonitoringgroup.Thisisusedtodefineaccesstothedevice’sMIBs.OntheCiscorouter,openaconsolesessionandgointoconfigurationmode.Thenissuethefollowingcommand:
R1(config)#snmp-servergroupMonitoringGroupv3priv
Thev3keywordspecifiesthatwewanttouseSNMPv3,whiletheprivkeywordspecifiesthatwewanttousebothauthenticationandencryption.It’spossibletopassmoreoptionstotheprecedingcommandinordertodefineanaccesslistifyouwanttolimitaccesstospecificMiBs,butwe’llkeepthingssimplehereandletourZabbixprobeaccessallMIBs.
Nowthatwehaveagroup,wecancreateauser,asfollows:
R1(config)#snmp-serveruserzabbixMonitoringGroupv3authshazbxpasspriv
aes128zbxpriv
Asyoucansee,weassignedtheZabbixusertothepreviouslycreatedgroupanddefinedtheauthenticationandencryptionpassphrases.Takenoteofalltheseelementsasyou’llneedtospecifyallofthemonZabbix’ssideandtheywillneedtomatchwhatyouusedhere.Tosummarize,hereiswhatyou’llinputlaterwhenconfiguringanSNMPv3Zabbixitem:
Field Value
User zabbix
Authenticationprotocol sha
Authenticationpassphrase zbxpass
Privacyprotocol aes
Privacypassphrase zbxpriv
NotePleasedon’tusethepassphrasesshownhere.Theseareintentionallyweak,andweusedthemforillustrationpurposesonly.
Thisisallthereistoit.Later,we’lladdsomeinformationabouttellingtheappliance
www.it-ebooks.info
wheretosendSNMPtraps,butfornowyou’rereadytogetSNMPvaluesfromyourappliance,solet’sfocusonthatforawhile.
FindingtherightOIDstomonitorWhileZabbix’sdefaultSNMPtemplateswillhelpyougetstartedwithbasicmonitoring,you’llsoonfindtheneedtopollyourdevicesformoreinformation.Todothat,you’llneedtoknowtheOIDofthemetricyouwanttomonitoraswellasthedatatypeitwillyield.Afirstoptionistoconsultyourvendor’sdocumentationonthedeviceandfindoutwhichMIBsandOIDsareexposedbytheSNMPagent.Another,moreinteractive,optionistofindthemusingthesnmpwalkutilityanddirectlyaskingyourdeviceforthem.
NoteIfyoudon’talreadyhavesnmpwalk(andtheotherSNMPutilitiesforLinux)installed,youcanquicklydosowithasimplecommand:
#yuminstallnet-snmp-utils
OIDsaresentandreceivedbySNMPagentsandserversasdottedsequencesofnumbers.JustlikeIPaddresses,thisisconvenientformachine-to-machinecommunication,buthardtoreadforhumans.Inordertomakethemostfromtheexplorationofyourdeviceusingsnmpwalk,makesureyouhavealltheMIBsyouneedinstalled.MIBsessentiallymapOIDstoreadableandunderstandabledescriptionsofthemselves.Inotherwords,theytakeoutputlikethisone:
.1.3.6.1.2.1.2.2.1.1.1=INTEGER:1
.1.3.6.1.2.1.2.2.1.1.2=INTEGER:2
.1.3.6.1.2.1.2.2.1.1.3=INTEGER:3
.1.3.6.1.2.1.2.2.1.1.5=INTEGER:5
.1.3.6.1.2.1.2.2.1.2.1=STRING:lo
.1.3.6.1.2.1.2.2.1.2.2=STRING:eth1
.1.3.6.1.2.1.2.2.1.2.3=STRING:tap0
.1.3.6.1.2.1.2.2.1.2.5=STRING:br0
.1.3.6.1.2.1.2.2.1.3.1=INTEGER:softwareLoopback(24)
.1.3.6.1.2.1.2.2.1.3.2=INTEGER:ethernetCsmacd(6)
.1.3.6.1.2.1.2.2.1.3.3=INTEGER:ethernetCsmacd(6)
.1.3.6.1.2.1.2.2.1.3.5=INTEGER:ethernetCsmacd(6)
.1.3.6.1.2.1.2.2.1.4.1=INTEGER:16436
.1.3.6.1.2.1.2.2.1.4.2=INTEGER:1500
.1.3.6.1.2.1.2.2.1.4.3=INTEGER:1500
.1.3.6.1.2.1.2.2.1.4.5=INTEGER:1500
.1.3.6.1.2.1.2.2.1.5.1=Gauge32:10000000
.1.3.6.1.2.1.2.2.1.5.2=Gauge32:1000000000
.1.3.6.1.2.1.2.2.1.5.3=Gauge32:10000000
.1.3.6.1.2.1.2.2.1.5.5=Gauge32:0
.1.3.6.1.2.1.2.2.1.6.1=STRING:
.1.3.6.1.2.1.2.2.1.6.2=STRING:0:c:29:24:15:50
.1.3.6.1.2.1.2.2.1.6.3=STRING:2:10:f7:72:77:50
.1.3.6.1.2.1.2.2.1.6.5=STRING:0:c:29:24:15:50
.1.3.6.1.2.1.2.2.1.7.1=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.7.2=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.7.3=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.7.5=INTEGER:up(1)
www.it-ebooks.info
.1.3.6.1.2.1.2.2.1.8.1=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.8.2=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.8.3=INTEGER:up(1)
.1.3.6.1.2.1.2.2.1.8.5=INTEGER:up(1)
Then,theyturnitintoamuchmorereadableform:
IF-MIB::ifIndex.1=INTEGER:1
IF-MIB::ifIndex.2=INTEGER:2
IF-MIB::ifIndex.3=INTEGER:3
IF-MIB::ifIndex.5=INTEGER:5
IF-MIB::ifDescr.1=STRING:lo
IF-MIB::ifDescr.2=STRING:eth1
IF-MIB::ifDescr.3=STRING:tap0
IF-MIB::ifDescr.5=STRING:br0
IF-MIB::ifType.1=INTEGER:softwareLoopback(24)
IF-MIB::ifType.2=INTEGER:ethernetCsmacd(6)
IF-MIB::ifType.3=INTEGER:ethernetCsmacd(6)
IF-MIB::ifType.5=INTEGER:ethernetCsmacd(6)
IF-MIB::ifMtu.1=INTEGER:16436
IF-MIB::ifMtu.2=INTEGER:1500
IF-MIB::ifMtu.3=INTEGER:1500
IF-MIB::ifMtu.5=INTEGER:1500
IF-MIB::ifSpeed.1=Gauge32:10000000
IF-MIB::ifSpeed.2=Gauge32:1000000000
IF-MIB::ifSpeed.3=Gauge32:10000000
IF-MIB::ifSpeed.5=Gauge32:0
IF-MIB::ifPhysAddress.1=STRING:
IF-MIB::ifPhysAddress.2=STRING:0:c:29:24:15:50
IF-MIB::ifPhysAddress.3=STRING:2:10:f7:72:77:50
IF-MIB::ifPhysAddress.5=STRING:0:c:29:24:15:50
IF-MIB::ifAdminStatus.1=INTEGER:up(1)
IF-MIB::ifAdminStatus.2=INTEGER:up(1)
IF-MIB::ifAdminStatus.3=INTEGER:up(1)
IF-MIB::ifAdminStatus.5=INTEGER:up(1)
IF-MIB::ifOperStatus.1=INTEGER:up(1)
IF-MIB::ifOperStatus.2=INTEGER:up(1)
IF-MIB::ifOperStatus.3=INTEGER:up(1)
IF-MIB::ifOperStatus.5=INTEGER:up(1)
IfyouhavetherightMIBs,youwon’thavetoguessthemeaningofeachOIDfromitsvalueasmostofthetime,itwillbeclearenoughfromitsname.ToaddanewMIBtoyourSNMPtools,youhavetoobtainitfromthevendorofyourdeviceandtheninstallitonyoursystem.VendorsusuallymaketheirMIBsfreelyavailable,soyoushouldn’thaveanyproblemsfindingthem.
HerearesomeofthemajorvendorsofMIBsources,compiledatthetimeofwriting:
Vendor MIBs
Cisco http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml
Juniper http://www.juniper.net/techpubs/software/index_mibs.html
Barracudanetworks https://techlib.barracuda.com/search/go/global?q=MIB
www.it-ebooks.info
NoteAveryusefulresourceisOIDView’sfreeMIBdatabasethatyoucanfindhere:
http://www.oidview.com/mibs/detail.html
Atthetimeofwritingthis,thedatabasehadmorethan7,000MIBs,sochancesareyou’llbeabletofindaMIBforthemostobscurenetworkdeviceyoumighthavetomonitor.
MIBsareplaintextfiles,soifyouhaveacompressedarchive,youwillneedtounpackitbeforeyoucaninstallitscontents.OnceyouhavetheplaintextMIBS,it’sasimplematterofcopyingtheminto/usr/share/snmp/mibsandthenusingthe-moptiontotheSNMPcommandstospecifywhichMIByouwanttoloadinadditiontothedefaultones.
ShouldyourMIBscollectionbecometoobigandyouwantedtoorganizethemindifferentdirectories,thenyou’llneedtotellyourtoolswheretofindthem.Youhavetwooptions:eitherspecifyfromthecommandlinethedirectoriesyouwantyourcommandtosearchforMIBs,orputthisinformationinaconfigurationfilesothatyourcommandsalwaysknowtheMIBs’location.Theoptionsarediscussedasfollows:
Thefirstoptionisusefulifyou’rejusttryingoutanewMIBandseeingwhetherthat’stheoneyouneed.EveryNet-SNMP-basedcommandwilltakea-moptionthatyoucanusetospecifyaspecificMIBtoloadfromthemibsdirectory.Here’sacommandforexample:
$snmpwalk-m+CISCO-STUN-MIB-v3-uzabbix-aSHA-Azbxpassword-l
AuthPriv-xAES-Xprivpassword10.10.1.9
ThiscommandwilluseSNMPv3tocontacttheSNMPagentat10.10.1.9withthespecifiedcredentialsandwillloadtheCISCO-STUN-MIBthatitwillfindinthe/usr/share/snmp/mibsdirectory,inadditiontothosealreadyloadedasdefault.
Thesecondoptionismorepermanentandinvolvesediting(orcreating,ifit’snotalreadythere)the/etc/snmp/snmp.conffile.JustaddalinewiththelistofdirectoriestosearchformibsandanotherlinethatspecifieswhichMIBsthecommandsshouldactuallyload(inthiscase,we’llloadallofthem),asfollows:
mibdirs
/usr/share/snmp/mibs:/usr/share/snmp/mibs/cisco:/usr/share/snmp/mibs/ju
niper:/mnt/remote/shared_mibs/
mibs+ALL
Asyoucansee,evenifyoukeepyoursubdirectoriesin/usr/share/snmp/mibs,you’llhavetospecifyeachoneyouwantautomaticallyincluded.OnceyouhaveyourMIBsinstalledandloaded,you’llbereadytofullyexploreyourdevices’SNMPagents.ToperformacompletesnmpwalkonadevicecantakequitealotoftimeandproducealotofoutputdependingonhowmanyOIDsitexposes.Aroutercanhavethousandsofthem,soit’sadvisabletoredirectthecommand’soutputtoafilesothatyouareabletoreferenceitandexploreitatanytimeyouwantwithouthavingtoperformacompletewalkonthedeviceitself,asfollows:
$snmpwalk-v3-uzabbix-aSHA-Azbxpassword-lAuthPriv-xAES-X
www.it-ebooks.info
privpassword10.10.1.9>router-R1-snmp_baseline.txt
AnotheradvantageofhavingtheMIBsyouneedisthatit’llbeeasiertocreatenewSNMPitemsinZabbixasyou’llbeabletospecifythestringversionofanOIDandnotonlyitsnumericalvalue.ZabbixreliesontheNet-SNMPlibrary,soitwillalsoreferenceanyMIBsinstalledinyoursystem’sdefaultdirectories.
Solet’sseehowyoucanusetheoutputofsnmpwalktocreatenewZabbixitems.
MappingSNMPOIDstoZabbixitemsAnSNMPvalueiscomposedofthreedifferentparts:theOID,thedatatype,andthevalueitself.WhenyouusesnmpwalkorsnmpgettogetvaluesfromanSNMPagent,theoutputlookslikethis:
SNMPv2-MIB::sysObjectID.0=OID:CISCO-PRODUCTS-MIB::cisco3640
DISMAN-EVENT-MIB::sysUpTimeInstance=Timeticks:(83414)0:13:54.14
SNMPv2-MIB::sysContact.0=STRING:
SNMPv2-MIB::sysName.0=STRING:R1
SNMPv2-MIB::sysLocation.0=STRING:Upperfloorroom13
SNMPv2-MIB::sysServices.0=INTEGER:78
SNMPv2-MIB::sysORLastChange.0=Timeticks:(0)0:00:00.00
...
IF-MIB::ifPhysAddress.24=STRING:c4:1:22:4:f2:f
IF-MIB::ifPhysAddress.26=STRING:
IF-MIB::ifPhysAddress.27=STRING:c4:1:1e:c8:0:0
IF-MIB::ifAdminStatus.1=INTEGER:up(1)
IF-MIB::ifAdminStatus.2=INTEGER:down(2)
…
Andsoon.
Thefirstpart,theonebeforethe=signis,naturally,theOID.ThiswillgointotheSNMPOIDfieldintheZabbixitemcreationpageandistheuniqueidentifierforthemetricyouareinterestedin.SomeOIDsrepresentasingleanduniquemetricforthedevice,sotheyareeasytoidentifyandaddress.Intheaboveexcerpt,onesuchOIDisDISMAN-EVENT-MIB::sysUpTimeInstance.IfyouareinterestedinmonitoringthatOID,you’donlyhavetofillouttheitemcreationformwiththeOIDitselfandthendefineanitemname,adatatype,andaretentionpolicy,andyouarereadytostartmonitoringit.Inthecaseofanuptimevalue,time-ticksareexpressedinseconds,soyou’llchooseanumericdecimaldatatype.We’llseeinthenextsectionhowtochooseZabbixitemdatatypesandhowtostorevaluesbasedonSNMPdatatypes.You’llalsowanttostorethevalueasisandoptionallyspecifyaunitofmeasure.Thisisbecauseanuptimeisalreadyarelativevalueasitexpressesthetimeelapsedsinceadevice’slatestboot.Therewouldbenopointincalculatingafurtherdeltawhengettingthismeasurement.Finally,you’lldefineapollingintervalandchoosearetentionpolicy.Inthefollowingexample,thepollingintervalisshowntobe5minutes(300seconds),thehistoryretentionpolicyas3days,andthetrendstorageperiodasoneyear.Theseshouldbesensiblevaluesasyoudon’tnormallyneedtostorethedetailedhistoryofavaluethateitherresetstozero,or,bydefinition,growslinearlybyonetickeverysecond.
Thefollowingscreenshotencapsulateswhathasbeendiscussedinthisparagraph:
www.it-ebooks.info
Rememberthattheitem’skeyvaluestillhastobeuniqueatthehost/templatelevelasitwillbereferencedtobyallotherZabbixcomponents,fromcalculateditemstotriggers,maps,screens,andsoon.Don’tforgettoputtherightcredentialsforSNMPv3ifyouareusingthisversionoftheprotocol.
ManyofthemoreinterestingOIDs,though,areabitmorecomplex:multipleOIDscanberelatedtooneanotherbymeansofthesameindex.Let’slookatanothersnmpwalkoutputexcerpt:
IF-MIB::ifNumber.0=INTEGER:26
IF-MIB::ifIndex.1=INTEGER:1
IF-MIB::ifIndex.2=INTEGER:2
IF-MIB::ifIndex.3=INTEGER:3
…
IF-MIB::ifDescr.1=STRING:FastEthernet0/0
www.it-ebooks.info
IF-MIB::ifDescr.2=STRING:Serial0/0
IF-MIB::ifDescr.3=STRING:FastEthernet0/1
…
IF-MIB::ifType.1=INTEGER:ethernetCsmacd(6)
IF-MIB::ifType.2=INTEGER:propPointToPointSerial(22)
IF-MIB::ifType.3=INTEGER:ethernetCsmacd(6)
…
IF-MIB::ifMtu.1=INTEGER:1500
IF-MIB::ifMtu.2=INTEGER:1500
IF-MIB::ifMtu.3=INTEGER:1500
…
IF-MIB::ifSpeed.1=Gauge32:10000000
IF-MIB::ifSpeed.2=Gauge32:1544000
IF-MIB::ifSpeed.3=Gauge32:10000000
…
IF-MIB::ifPhysAddress.1=STRING:c4:1:1e:c8:0:0
IF-MIB::ifPhysAddress.2=STRING:
IF-MIB::ifPhysAddress.3=STRING:c4:1:1e:c8:0:1
…
IF-MIB::ifAdminStatus.1=INTEGER:up(1)
IF-MIB::ifAdminStatus.2=INTEGER:down(2)
IF-MIB::ifAdminStatus.3=INTEGER:down(2)
…
IF-MIB::ifOperStatus.1=INTEGER:up(1)
IF-MIB::ifOperStatus.2=INTEGER:down(2)
IF-MIB::ifOperStatus.3=INTEGER:down(2)
…
IF-MIB::ifLastChange.1=Timeticks:(1738)0:00:17.38
IF-MIB::ifLastChange.2=Timeticks:(1696)0:00:16.96
IF-MIB::ifLastChange.3=Timeticks:(1559)0:00:15.59
…
IF-MIB::ifInOctets.1=Counter32:305255
IF-MIB::ifInOctets.2=Counter32:0
IF-MIB::ifInOctets.3=Counter32:0
…
IF-MIB::ifInDiscards.1=Counter32:0
IF-MIB::ifInDiscards.2=Counter32:0
IF-MIB::ifInDiscards.3=Counter32:0
…
IF-MIB::ifInErrors.1=Counter32:0
IF-MIB::ifInErrors.2=Counter32:0
IF-MIB::ifInErrors.3=Counter32:0
…
IF-MIB::ifOutOctets.1=Counter32:347968
IF-MIB::ifOutOctets.2=Counter32:0
IF-MIB::ifOutOctets.3=Counter32:0
Asyoucansee,foreverynetworkinterface,thereareseveralOIDs,eachonedetailingaspecificaspectoftheinterface:itsname,itstype,whetherit’supordown,theamountoftrafficcominginorgoingout,andsoon.ThedifferentOIDsarerelatedthroughtheirlastnumber,theactualindexoftheOID.Lookingattheprecedingexcerpt,weknowthatthedevicehas26interfaces,ofwhichweareshowingsomevaluesforjustthefirstthree.Bycorrelatingtheindexnumbers,wealsoknowthatinterface1iscalledFastEthernet0/0,itsMACaddressisc4:1:1e:c8:0:0,theinterfaceisupandhasbeenupforjust17
www.it-ebooks.info
seconds,andsometrafficalreadywentthroughit.
Now,onewaytomonitorseveralofthesemetricsforthesameinterfaceistomanuallycorrelatethesevalueswhencreatingtheitems,puttingthecompleteOIDintheSNMPOIDfield,andmakingsurethatboththeitemkeyanditsnamereflecttherightinterface.Thisprocessisnotonlypronetoerrorsduringthesetupphase,butitcouldalsointroducesomeinconsistenciesdowntheroad.Thereisnoguarantee,infact,thattheindexwillremainconsistentacrosshardwareorsoftwareupgradesorevenacrossconfigurationswhenitcomestomorevolatilestateslikethenumberofVLANsorroutingtablesinsteadofnetworkinterfaces.FortunatelyZabbixprovidesafeature,calleddynamicindexes,thatallowsyoutoactuallycorrelatedifferentOIDsinthesameSNMPOIDfieldsothatyoucandefineanindexbasedontheindexexposedbyanotherOID.
ThismeansthatifyouwanttoknowtheadminstatusofFastEthernet0/0,youdon’tneedtofindtheindexassociatedwithFastEthernet0/0(inthiscaseitwouldbe1)andthenaddthatindextoIF-MIB::ifAdminStatusofthebaseOID,hopingthatitwon’teverchangeinthefuture.Youcaninsteadusethefollowingcode:
IF-MIB::ifAdminStatus["index","IF-MIB::ifDescr","FastEthernet0/0"]
UponusingtheprecedingcodeintheSNMPOIDfieldofyouritem,theitemwilldynamicallyfindtheindexoftheIF-MIB::ifDescrOIDwherethevalueisFastEthernet0/0andappendittoIF-MIB::ifAdminStatusinordertogettherightstatusfortherightinterface.
Ifyouorganizeyouritemsthisway,you’llalwaysbesurethatrelateditemsactuallyshowtherightrelatedvaluesforthecomponentyouareinterestedinandnotthoseofanotheronebecausethingschangedonthedevice’ssidewithoutyourknowledge.Moreover,we’llbuildonthistechniquetodeveloplow-leveldiscoveryofadeviceaswe’llseeinChapter4,DiscoveringYourNetwork.
Youcanusethesametechniquetogetotherinterestinginformationoutofadevice.Consider,forexample,thefollowingexcerpt:
ENTITY-MIB::entPhysicalVendorType.1=OID:CISCO-ENTITY-VENDORTYPE-OID-
MIB::cevChassis3640
ENTITY-MIB::entPhysicalVendorType.2=OID:CISCO-ENTITY-VENDORTYPE-OID-
MIB::cevContainerSlot
ENTITY-MIB::entPhysicalVendorType.3=OID:CISCO-ENTITY-VENDORTYPE-OID-
MIB::cevCpu37452fe
ENTITY-MIB::entPhysicalClass.1=INTEGER:chassis(3)
ENTITY-MIB::entPhysicalClass.2=INTEGER:container(5)
ENTITY-MIB::entPhysicalClass.3=INTEGER:module(9)
ENTITY-MIB::entPhysicalName.1=STRING:3745chassis
ENTITY-MIB::entPhysicalName.2=STRING:3640ChassisSlot0
ENTITY-MIB::entPhysicalName.3=STRING:c3745MotherboardwithFast
EthernetonSlot0
ENTITY-MIB::entPhysicalHardwareRev.1=STRING:2.0
ENTITY-MIB::entPhysicalHardwareRev.2=STRING:
www.it-ebooks.info
ENTITY-MIB::entPhysicalHardwareRev.3=STRING:2.0
ENTITY-MIB::entPhysicalSerialNum.1=STRING:FTX0945W0MY
ENTITY-MIB::entPhysicalSerialNum.2=STRING:
ENTITY-MIB::entPhysicalSerialNum.3=STRING:XXXXXXXXXXX
Itshouldbeimmediatelycleartoyouthatyoucanfindthechassis’sserialnumberbycreatinganitemwith:
ENTITY-MIB::entPhysicalSerialNum["index","ENTITY-MIB::entPhysicalName",
"3745chassis"]
Thenyoucanspecify,inthesameitem,thatitshouldpopulatetheSerialNumberfieldofthehost’sinventory.Thisishowyoucanhaveamoreautomatic,dynamicpopulationofinventoryfields.
Thepossibilitiesareendlessaswe’veonlyjustscratchedthesurfaceofwhatanygivendevicecanexposeasSNMPmetrics.BeforeyougoandfindyourfavoriteOIDstomonitorthough,let’shaveacloserlookattheprecedingexamples,andlet’sdiscussdatatypes.
GettingdatatypesrightWehavealreadyseenhowanOID’svaluehasaspecificdatatypethatisusuallyclearlystatedwiththedefaultsnmpwalkcommand.Intheprecedingexamples,youcanclearlyseethedatatypejustafterthe=sign,beforetheactualvalue.ThereareanumberofSNMPdatatypes—somestillcurrentandsomedeprecated.YoucanfindtheofficiallistanddocumentationinRFC2578(http://tools.ietf.org/html/rfc2578),butlet’shavealookatthemostimportantonesfromtheperspectiveofaZabbixuser:
SNMPtype Description SuggestedZabbixitemtypeandoptions
INTEGERThiscanhavenegativevaluesandisusuallyusedforenumerations
Numericunsigned,decimalStorevalueasisShowwithvaluemappings
STRING Thisisaregularcharacterstringandcancontainnewlines TextStorevalueasis
OID ThisisanSNMPobjectidentifier CharacterStorevalueasis
IpAddress IPv4onlyCharacterStorevalueasis
Counter32 Thisincludesonlynon-negativeandnondecreasingvaluesNumericunsigned,decimalStorevalueasdelta(speedpersecond)
Gauge32 Thisincludesonlynon-negativevalues,whichcandecrease Numericunsigned,decimalStorevalueasis
www.it-ebooks.info
Counter64 Thisincludesnon-negativeandnondecreasing64-bitvalues Numericunsigned,decimalStorevalueasdelta(speedpersecond)
TimeTicks Thisincludesnon-negative,nondecreasingvalues Numericunsigned,decimalStorevalueasis
Firstofall,rememberthattheabovesuggestionsarejustthat—suggestions.Youshouldalwaysevaluatehowtostoreyourdataonacase-by-casebasis,butyou’llprobablyfindthatinmanycasesthoseareindeedthemostusefulsettings.
Movingontotheactualdatatypes,rememberthatthecommandlineSNMPtoolsbydefaultparsethevaluesandshowsomealreadyinterpretedinformation.ThisisespeciallytrueforTimeticksvaluesandforINTEGERvalueswhentheseareusedasenumerations.Inotherwords,youseethefollowingfromthecommandline:
VRRP-MIB::vrrpNotificationCntl.0=INTEGER:disabled(2)
However,whatisactuallypassedasarequestisthebareOID:
1.3.6.1.2.1.68.1.2.0
TheSNMPagentwillrespondwithjustthevalue,which,inthiscase,isthevalue2.
Thismeansthatinthecaseofenumerations,Zabbixwilljustreceiveandstoreanumberandnotthestringdisabled(2)asseenfromthecommandline.Ifyouwanttodisplaymonitoringvaluesthatareabitclearer,youcanapplyvaluemappingstoyournumericitems.Valuemapscontainthemappingbetweennumericvaluesandarbitrarystringrepresentationsforahuman-friendlyrepresentation.Youcanspecifywhichoneyouneedintheitemconfigurationform,asfollows:
www.it-ebooks.info
Zabbixcomeswithafewpredefinedvaluemappings.Youcancreateyourownmappingsbyfollowingtheshowvaluemappingslinkand,providedyouhaveadminrolesonZabbix,you’llbetakentoapagewhereyoucanconfigureallvaluemappingsthatwillbeusedbyZabbix.Fromthere,clickonCreatevaluemapintheupper-rightcornerofthepage,andyou’llbeabletocreateanewmapping.NotallINTEGERvaluesareenumerations,butthosethatareusedassuchwillbeclearlyrecognizablefromyourcommand-linetoolsastheywillbedefinedasINTEGERvaluesbutwillshowastringlabelalongwiththeactualvalue,justasintheprecedingexample.
Ontheotherhand,whentheyarenotusedasenumerations,theycanrepresentdifferentthingsdependingonthecontext.Asseeninthepreviousparagraph,theycanrepresentthenumberofindexesavailableforagivenOID.Theycanalsorepresentapplicationorprotocol-specificvalues,suchasdefaultMTU,defaultTTL,routemetrics,andsoon.
Themaindifferencebetweengauges,counters,andintegersisthatintegerscanassumenegativevalues,whilegaugesandcounterscannot.Inadditiontothat,counterscanonlyincreaseorwraparoundandstartagainfromthebottomoftheirvaluerangeoncetheyreachtheupperlimitsofit.FromtheperspectiveofZabbix,thismarksthedifferenceinhowyou’llwanttostoretheirvalues.
Gaugesareusuallyemployedwhenavaluecanvarywithinagivenrange,suchasthespeedofaninterface,theamountoffreememory,oranylimitsandtimeoutsyoumightfindfornotifications,thenumberofinstances,andsoon.Inallofthesecases,thevaluecanincreaseordecreaseintime,soyou’llwanttostorethemastheyarebecauseonceputonagraph,they’lldrawameaningfulcurve.
Counters,ontheotherhand,canonlyincreasebydefinition.Theyaretypicallyusedtoshowhowmanypacketswereprocessedbyaninterface,howmanyweredropped,howmanyerrorswereencountered,andsoon.Ifyoustorecountervaluesastheyare,you’llfindinyourgraphssomeever-ascendingcurvesthatwon’ttellyouverymuchforyourmonitoringorcapacityplanningpurposes.Thisiswhyyou’llusuallywanttotrackacounter’samountofchangeintime,morethanitsactualvalue.Todothat,Zabbixofferstwodifferentwaystostoredeltasordifferencesbetweensuccessivevalues.
Thedelta(simplechange)storagemethoddoesexactlywhatitsays:itsimplycomputesthedifferencebetweenthecurrentlyreceivedvalueandthepreviouslyreceivedone,andstorestheresult.Itdoesn’ttakeintoconsiderationtheelapsedtimebetweenthetwomeasurements,northefactthattheresultcanevenhaveanegativevalueifthecounteroverflows.Thefactisthatmostofthetime,you’llbeveryinterestedinevaluatinghowmuchtimehaspassedbetweentwodifferentmeasurementsandintreatingcorrectlyanynegativevaluesthatcanappearasaresult.
Thedelta(speedpersecond)willdividethedifferencebetweenthecurrentlyreceivedvalueandthepreviouslyreceivedonebythedifferencebetweenthecurrenttimestampandthepreviousone,asfollows:
(value–prev_value)/(time-prev_time)
www.it-ebooks.info
Thiswillensurethatthescaleofthechangewillalwaysbeconstant,asopposedtothescaleofthesimplechangedelta,whichwillvaryeverytimeyoumodifytheupdateintervaloftheitem,givingyouinconsistentresults.Moreover,thespeed-per-seconddeltawillignoreanynegativevaluesandjustwaitforthenextmeasurement,soyouwon’tfindanyfalsedipsinyourgraphduetooverflowing.
Finally,whileSNMPusesspecificdatatypesforIPaddressesandSNMPOIDs,therearenosuchtypesinZabbix,soyou’llneedtomapthemtosomekindofstringitem.Thesuggestedtypehereischaracterasbothvalueswon’tbebiggerthan255charactersandwon’tcontainanynewlines.
Stringvalues,ontheotherhand,canbequitelongastheSNMPspecificationallowsfor65,535-character-longtexts;however,textthatlongwouldbeoflittlepracticalvalue.Eveniftheyareusuallymuchshorter,stringvaluescanoftencontainnewlinesandbelongerthan255characters.
Consider,forexample,thefollowingSysDescrOIDforthisdevice:
NMPv2-MIB::sysDescr.0=STRING:CiscoIOSSoftware,3700Software(C3745-
ADVENTERPRISEK9_SNA-M),Version12.4(15)T14,RELEASESOFTWARE(fc2)^M
TechnicalSupport:http://www.cisco.com/techsupport^M
Copyright(c)1986-2010byCiscoSystems,Inc.^M
CompiledTue17-Aug-1012:56byprod_rel_tea
Asyoucansee,thestringspansmultiplelines,andit’sdefinitelylongerthan255characters.Thisiswhythesuggestedtypeforstringvaluesistextasitallowstextofarbitrarylengthandstructure.Ontheotherhand,ifyou’resurethataspecificOIDvaluewillalwaysbemuchshorterandsimpler,youcancertainlyusethecharacterdatatypeforyourcorrespondingZabbixitem.
Now,youaretrulyreadytogetthemostoutofyourdevices’SNMPagentsasyouarenowabletofindtheOIDyouwanttomonitorandmapthemperfectlytoZabbixitems,downtohowtostorethevalues,theirdatatypes,withwhatfrequency,andwithanyvaluemappingthatmightbenecessary.
It’snowtimetoexploretheotheraspectofSNMP:traps.
www.it-ebooks.info
SNMPtrapsSNMPtrapsareabitofanoddballwhencomparedtoalltheotherZabbixitemtypes.Unlikeotheritems,SNMPtrapsdonotreportasimplemeasurement,butaneventofsometype.Inotherwords,theyaretheresultofsomekindofcheckorcomputationmadebytheSNMPagentandsentovertothemonitoringserverasastatusreport.AnSNMPtrapcanbeissuedeverytimeahostisrebooted,aninterfaceisdown,adiskisdamaged,oraUPShaslostpowerandiskeepingserversupusingitsbattery.
ThiskindofinformationcontrastswithZabbix’sbasicassumptionthatanitemisasimplemetricnotdirectlyrelatedtoaspecificevent.Ontheotherhand,there’snootherwaytobeawareofcertainsituationsifnotthroughanSNMPtrapeitherbecausetherearenorelatedmetrics(consider,forexample,theeventtheserverisbeingshutdown)orbecausetheappliance’sonlywaytoconveyitsstatusisthroughabunchofSNMPobjectsandtraps.
SotrapsareofrelativelylimitedusetoZabbixasyoucan’tdomuchmorethanbuildasimpletriggeroutofeverytrapandthennotifyabouttheevent(notmuchpointingraphingatraporbuildingcalculateditemsonit).Nevertheless,theymightproveessentialforacompletemonitoringsolution.
TomanageSNMPtrapseffectively,Zabbixneedsacoupleofhelpertools:thesnmptrapddaemontoactuallyhandleconnectionsfromtheSNMPagentsandsomekindofscripttocorrectlyformateverytrapandpassittotheZabbixserverforfurtherprocessing.
SnmptrapdIfyouhavecompiledSNMPsupportintotheZabbixserver,youshouldalreadyhavethecompleteSNMPsuiteinstalled,whichcontainstheSNMPdaemonandtheSNMPtrapdaemonalongwiththeutilitieswehaveusedintheprevioussection.
JustastheZabbixserverhasabunchofdaemonprocessesthatlistenonTCPport10051forincomingconnections(fromagents,proxies,andnodes),snmptrapdisthedaemonprocessthatlistensonUDPport162forincomingtrapscomingfromremoteSNMPagents.
Onceinstalled,snmptrapdreadsitsconfigurationoptionsfromansnmptrapd.conffilethatcanbeusuallyfoundinthe/etc/snmp/directory.ThebareminimumconfigurationforsnmptrapdrequiresthedefinitionofauserandaprivacylevelforSNMPv3,asfollows:
createUserzbxuserSHAauthAESpriv
authUserlog,execute,netzbxuser
TipTheaboveconfigurationwillenablesnmptrapdtoreceiveSNMPv3INFORMpackets.ThesearejustlikeregularSNMPtraps,withtwodifferences:thefirstoneisthatwhileanagentwon’texpectaresponseaftersendingatrap,INFORMpacketsareacknowledged,sosnmptrapdwillsendaresponseforeverytrapreceived.Butthemostimportantdifference
www.it-ebooks.info
isthatwithINFORMpackets,theauthoritativeEngineIDwillbethatofthereceivingpartyandnotthesendingpartyaswithregulartraps.Thismeansthatyou’llhavetospecifyyourserver’sEngineIDtoeverydevicethatwillsendSNMPv3INFORMpackets.Sinceyou’llhavetoconfigurethemtosendpacketstotheserveranyway,thiswon’tmeantoomuchwork.Manyagentsautomaticallydiscoverapeer’sEngineIDbeforesendinganINFORM,butifyouneedtosetityourself,youcandiscoveryourserver’sEngineIDusingsnmpgetandaskingforthesnmpEngineID.0OID.
IfyouwanttouseregularSNMPtraps,you’llhavetoinsertanewcreateUserlineforeveryagentthatwillsendtrapstotheserver,witheachonespecifyingthecorrectEngineIDoftheagentsendingtraps.
Withthisminimalconfiguration,snmptrapdwilllimititselftologthetraptosyslog.WhileitcouldbepossibletoextractthisinformationandsendittoZabbix,it’seasiertotellsnmptrapdhowitshouldhandletraps.Whilethedaemonhasnoprocessingcapabilitiesofitsown,itcanexecuteanycommandorapplicationeitherusingthetrapHandledirective,orleveragingitsembeddedPerlfunctionality.Thelatterismoreefficientasthedaemonwon’thavetoforkanewprocessandwaitforitsexecutiontofinish,soit’stherecommendedoneifyouplantoreceiveasignificantnumberoftraps.Justaddthefollowinglinetosnmptrapd.conf:
perldo"/usr/local/bin/zabbix_trap_receiver.pl";
TipYoucangetthezabbix_trap_receiverscriptfromtheZabbixsources.It’slocatedinmisc/snmptrap/zabbix_trap_receiver.pl.
BesuretocheckthatyoualsohavetheNet-SNMPPerlmoduleinstalled.Ifyouneedit,asimpleyuminstallnet-snmp-perlcommandshouldtakecareofeverything.
Oncerestarted,thesnmptrapddaemonwillexecutethePerlscriptyouspecifiedtoprocesseverytrapreceived,translatingitintoaformatthatcanbeeasilyparsedbytheZabbixserver.Inthefollowingsection,we’llseehowanSNMPtrapistranslatedandusedbyZabbix.
TransformingatrapintoaZabbixitemThePerlscriptincludedintheZabbixdistributionworksasatranslatorfromanSNMPtrapformattoaZabbixitemmeasurement.Foreverytrapreceived,itwillformatitaccordingtotherulesdefinedinthescriptandwilloutputtheresultinalogfile.Bydefault,thelogfileiscalled/tmp/zabbix_traps.tmp.YouneedtomakesurethatthesamefileisreadbyZabbixbysettingthefollowingparametersin/etc/zabbix/zabbix_server.conf:
###Option:StartSNMPTrapper
#If1,SNMPtrapperprocessisstarted.
#
#Mandatory:no
#Range:0-1
#Default:
www.it-ebooks.info
StartSNMPTrapper=1
###Option:SNMPTrapperFile
#TemporaryfileusedforpassingdatafromSNMPtrapdaemontothe
server.
#Mustbethesameasinzabbix_trap_receiver.plorSNMPTT
configurationfile.
SNMPTrapperFile=/tmp/zabbix_traps.tmp
Thelogfilewillhaveaformatsimilartothefollowingexample:
03:47:102014/12/09ZBXTRAP127.0.0.1
PDUINFO:
notificationtypeTRAP
version0
receivedfromUDP:[127.0.0.1]:34373->[127.0.0.1]
errorstatus0
messageid0
communitypublic
transactionid3
errorindex0
requestid0
VARBINDS:
DISMAN-EVENT-MIB::sysUpTimeInstancetype=67value=Timeticks:(55)
0:00:00.55
SNMPv2-MIB::snmpTrapOID.0type=6value=OID:IF-MIB::linkDown.0.33
IF-MIB::linkDowntype=4value=Hex-STRING:E2809C5445
53544D454E4F57E2809D
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0type=4value=STRING:"public"
SNMPv2-MIB::snmpTrapEnterprise.0type=6value=OID:IF-MIB::linkDown
TheZBXTRAPfollowedbytheIPaddresswillmarkthestartofanewlogstanza.Therestofthelogwillcontainalldetailsaboutthetrap,soyou’llbeabletoactonanyofthose.
TheZabbixserverwillinturnmonitortheaforesaidlogfileandprocesseverynewlineasanSNMPtrapitem,basicallymatchingthecontentofthelogtoanytrapitemdefinedfortherelevanthost.
Asyou’vealreadyseen,thefirstpartoftheloglineisusedbytheZabbixtrapreceivertomatchatrapwithitscorrespondinghost.Therestismatchedtotheaforesaidhost’sSNMPtrapitem’sregexpdefinitionsanditscontentaddedtoeverymatchingitem’shistoryofvalues.ThismeansthatifyouwishtohavealinkDowntrapitemforagivenhost,you’llneedtoconfigureanSNMPtrapitemwithansnmptrap["linkDown"]key,asfollows:
www.it-ebooks.info
Youmightneedtomakesurethatthelogtimeformatyouspecifyintheitem’sconfigurationwillmatchtheoneusedbythePerlscript.You’llalsohavetocheckthatthehost’sinterfacewillmatchtheoneloggedbysnmptrapdbecauseit’stheonepieceofdataZabbixwillusetomatchtrapstohosts.
Fromnowon,you’llbeabletoseethecontentsofthetrapintheitem’sdatahistory.
MovingonfromSNMP,therearestillotherdatasourcesthatyoucanrelyontogetmonitoringdataintoZabbix;forthepurposesofthisbook,themostinterestingonesarelogfiles.ComparedtoSNMP,theycanbetrickytoworkwith,buttheydohavetheiruses,solet’sexplorethemforawhile.
GettingnetflowfromthedevicestothemonitoringserverNetflowisaprotocoloriginallydevelopedbyCiscotocollectandmonitorstatisticsofnetworktrafficonadevice.Aftertheinitialrelease,manyvendorsstartedprovidingtheirownimplementationoftheprotocol.In2008IETFstandardizednetflowandpublishedInternetProtocolFlowInformationeXport(IPFIX)basedonnetflowv9withsomeextensions.However,netflowsomehowremainstheexistingnameoftheprotocolinfactbutnotnecessarilybylegalright,sothat’stheonewe’llusehere.
Anetflowrecordcontainsinformationaboutasinglenetworkflow.Aflowisasequenceofpacketsthatsharesomecommonproperties:
IPprotocolSourceIPaddressSourceport(forTCPandUDP)DestinationIPaddressDestinationport(forTCPandUDP)InputinterfaceTypeofservice
Foreachflow,arecordexposesmanydifferentvalues,whichchangewithnetflow
www.it-ebooks.info
versionsandimplementations.Herearethemostcommonones:
InputinterfaceofthedeviceOutputinterfaceofthedeviceFlowstarttimeFlowendtimeNumberofbytesintheflowNumberofpacketsintheflowSourceIPaddressSourceIPportSourceIPmaskDestinationIPaddressDestinationIPportDestinationIPmaskICMPtypeandcodeTCPflagsIPaddressoftheimmediatenext-hop
Itshouldbeimmediatelycleartoyouthatthistypeofinformationcanbeextremelyusefultoanetworkadministratorasitallowsyoutobuildapictureofallthetraffictraversingyournetwork.ItcanalsobeusedtoidentifyanomaloustrafficandtraffictoandfromIPaddressesorportsthatshouldnotbethere,orasforensicevidenceafteranincident.Moreover,itcanbeusedasasourceforcapacity-planninganalysistoidentifybottlenecksinyournetwork,periodsofpeakuse,andtoptalkersamongyourserversanddevices.
Finally,aswewereexplainingpreviously,it’sagoodcandidateforaZabbixlogitemasflowdataisusefulevenifitisnotdirectlyrelatedtothehostthatgeneratedit(evenifit’sstillusefultotrackthatpieceofinformationwheneverpossible).
So,let’sseehowtogetnetflowdataintoZabbix.
Firstofall,you’llhavetoconfigureyourdevicetosendflowdatatoaserver.InthecaseofaCiscodevice,herearetheconfigurationcommandsthatyouneedtoissue(remembertosubstituteallreferencestotheexampleZabbixserverwiththerealonesthatapplytoyourenvironment):
R1(config)#ipflow-exportdestination192.168.234.1319995
R1(config)#ipflow-exportversion9
R1(config)#interfacef0/0
R1(config-if)#ipflowingress
R1(config-if)#ipflowegress
R1(config-if)#exit
Inthefirstline,wespecifytheIPaddressofourZabbixserverandtheUDPportthedeviceshouldsendnetflowinformationto.
Thesecondlinesetsthenetflowversion.
Inthethirdline,wegointointerfacef0/0mode.Pleasenotethatyou’llhavetoexplicitlyenablenetflowforeveryinterfaceyouareinterestedin.Thisisusuallynotaproblem
www.it-ebooks.info
becauseifyouconfigurenetflowontherightinterfacesofyourrouters,you’llseemost,ifnotallofyourtrafficanyway;youwon’tneedtoenablenetflowoneveryinterfaceofeverynetworkdeviceyouhave.
Thefourthlineenablesnetflowmonitoringforincomingtrafficoninterfacef0/0,whilethefifthlineenablesnetflowmonitoringforoutgoingtrafficonthesameinterface.Ifyouwanttoenablenetflowonotherinterfaces,you’llneedtorepeatlines3to5foreveryinterfaceyouareinterestedin.
Repeatthewholeprocessforalltheroutersyouwanttogetflowinformationfrom,andonceyouaredone,youarereadytoturntoyourZabbixserver.
ReceivingnetflowdataonyourserverToactuallyreceiveandprocessnetflowpacketsonaserver,youneedadaemonthatwilllistenonaspecifiedUDPport,andthatwillunderstandthenetflowprotocol.OnLinux,suchdaemonsandassociatedtoolsarecontainedinthenfdumppackage.
Nfdumpisacollectionoftoolsthatwillenableyoutocapturenetflowdata,storeitondisk,filterit,andanalyzeit.Themostimportantcomponentsare:
nfcapd:Thisisthedaemoncomponentthatlistensforincomingnetflowdataandstoresitondiskinbinaryformatnfdump:Thisissimilartotcpdump;itreadsandfiltersnfcapdfiles,andoutputsreadabledata
Sothebasicdataflowwillbesimilartothisone:
1. Aroutersendsnetflowdatatotheserver.2. Ontheserver,nfcapdcapturesthedataandstoresitinbinaryfiles.3. Aschedulednfdumpprocesswillreadthebinaryfilesandpopulateahumanreadable
logwithnetflowinformation.4. AZabbixagentwillreadthelogandsenddatatotheZabbixserveraccordingtothe
item’sconfiguration.
Wehavealreadytakencareofpoint1,solet’sseehowtoinstallandconfigurethenfdumppackage,beforelookingintotheZabbixside.
Unfortunately,therearenoreadymaderpmpacketsfornfdump,sowe’llneedtofindthesourcecode,compileit,andinstallit.Thisisusuallyastraightforwardprocess.Firstofall,let’sinstallsomerequireddependenciesfornfdump:
#yuminstallrrdtoolrrdtool-develrrdtool-docperl-rrdtool
Then,we’llneedtodownloadthelatestsources.Atthemomentofwritingthis,thelatestavailableversionis1.6.12.Youcandownloadthepackagefromhttp://sourceforge.net/projects/nfdump/andthentransferittoyourserver.Onceyouhavetar.gzready,unpackit:
$tarxvzfnfdump-1.6.12.tar.gz
www.it-ebooks.info
Thenmoveintothenfdump-1.6.12directoryandruntheusualconfigure,make,andmakeinstallsequence.Ifyouwanttoinstallnfdumpinthemaindirectoriesinsteadofthe/usr/localtree,justpassthe–prefixoptiontotheconfigurescript.Inthefollowingexample,that’swhatwe’lluse:
$cdnfdump-1.6.12
$./configure–prefix=/usr--sysconfdir=/etc
$make
$suroot
#makeinstall
Onceinstalled,youcanaddadedicateduserfornfcapdsothatitdoesn’thavetorunasrootandsetaworkingdirectoryforit:
#useradd-s/sbin/nologinnetflow
#mkdir-p/var/nfdump/nfcapd
#mkdir-p/var/nfdump/logs
#chown-Rnetflow/var/nfdump
Whenyourunnfcapd,itwillcreateitsbinaryfilesunder/var/nfdump/nfcapd.Nfcapdfilesarerotated,bydefault,onceeveryfiveminutesandcanbeseparatedintoonedumpcollection(currentandrotatedfiles)persendinghostorasinglecollectionforallsendinghosts.Theycanalsobeexpiredafterasetamountoftime.Youarenowreadytowaitfornetflowdataandtransformitintoalogfile.Todothat,you’llneedtopasstherightoptiontonfcapd.Sincetherearequiteafewoptionstopass,let’sbuildthecommandlinelittlebylittle.Pleasedon’truntheintermediatecommands,butonlythefinalone;nfcapdwillcomplainaboutmissingoptionsandrefusetorun.
Firstofall,we’llpasssomeoptionsthatwillinstructnfcapdtogointodaemonmode(-D),tocompressoutput(-z),torunasusernetflow(-u),andtolistenonport9995(-p):
#nfcapd-D-z-unetflow-p9995
Then,we’llneedtoaddsomeoptionsaboutdatasources.Theacceptedcurrentmethodistousethe-nswitch.We’llalsoinstructnfcapdtocreateadditionalsubdirectoriestostorethecapfilestobetterorganizethem(-S):
#nfcapd-D-z-unetflow-p9995-nR1,192.168.11.9,/var/nfdump/nfcapd-n
R2,10.10.1.254,/var/nfdump/nfcapd-S2
Asyoucansee,you’llhavetospecifyadifferent-noptionforeverysourceyouconfigure.Ifyouhavemanynetflowsources,itmightbebettertorundifferentinstancesofnfcapdondifferentUDPportssoastosharetheloadbetweendifferentprocesses.Inthatcase,justremembertoconfigureyourdevicesaccordinglysothattheysendtheirtraffictothecorrectUDPport.The-S2optionwillcreateadditionalyear/month/day/hourdirectoriesunder/var/nfdump/nfcapdtostorecurrentandrotatedfiles.
Nfcapdfilesarerotatedeveryfiveminutes,andifyournetworkhasalotoftraffic,yournfcapddirectorycanbecomehuge.Youcouldscheduleaseparatejobtocleanthemup,butwiththe-eoption,nfcapdwillbeabletoalsotakecareofthat.Justsettheexpirationparameterwithnfexpireandnfcapdwillpickthemup:
www.it-ebooks.info
#nfexpire-u/var/nfdump/nfcapd-s15G-t90d
#nfcapd-D-z-unetflow-p9995-nR1,192.168.11.9,/var/nfdump/nfcapd-n
R2,10.10.1.254,/var/nfdump/nfcapd-S2-e
Intheaboveexample,wesetthesizelimitofthedirectoryto15gigabytes,andthecap(maximum)fileageto90days.Fileswillbedeletedbynfcapdwheneveroneoftheselimitsisreached.Thelastlineintheprecedingcommandnowcontainsalltheparametersweneedforbasicnetflowdumping.Ifyourunit(don’tforgetthenfexpirecommandtoo)orputitintoastartupscript,nfcapdwilllistenonthespecifiednetworkportforincomingnetflowdataandwriteittothedirectoriesyouspecified.
Onceyouhavesomedatain,youcanreaditwithnfdumpandoutputahuman-readablesetofrecords:
$nfdump-r/var/nfdump/nfcapd/2014/10/29/02/nfcapd.201410290250-o
extended
DateflowstartDurationProtoSrcIPAddr:PortDstIP
Addr:PortFlagsTosPacketsBytesppsbpsBppFlows
2014-10-2902:51:53.16063.545TCP10.13.27.151:80->
123.43.98.124:6523.AP.SF01288412055056
1
2014-10-2902:53:13.37023.135TCP64.76.73.121:25->
10.138.41.151:7643.AP.SF0512450055156
1
...
Timewindow:Oct29201402:50:00-Oct29201402:54:56
Thisisgettingclosertoourobjective.Ifyourunnfdumpandredirectitsoutputtoafileinsteadofthescreen,thereyouhavethelogfilewe’vebeentalkingaboutinthelastseveralpages.Todothat,youareprobablythinkingofsettingupacronjobthatwillfindthelatestnfcapdfilesthatweren’talreadyparsedbynfdump,makenfdumpreadthemwhilespecifyingatimewindowsothatyourlogfilewon’tcontainduplicateddata,andaddtheaforesaidoutputtoalogfilethatwillbemonitoredbyZabbix.Thiscanbeanontrivialexercisewhenyouconsiderthatnfcapdwillcontinuallyproducenewfilesandwillputtheminnewdirectoriesallthetime.Moreover,you’llneedtokeepsomekindofexecutionstatewiththetimestampofthelasttimenfdumpwasruninordertoavoidtheaforesaidduplicates.
Itturnsoutthatyou’llbeabletoavoidallthiswork,thankstoaniceoptionfornfcapd,the-xoption.Solet’srewritethenfcapdcommandonelasttime:
#nfcapd-D-z-unetflow-p9995-nR1,192.168.11.9,/var/nfdump/nfcapd-n
R2,10.10.1.254,/var/nfdump/nfcapd-S2-e-x'nfdump-q-oextended-r%d/%f
>>/var/nfdump/logs/zabbix_netflow.log'
The-xcommandexecutesanarbitrarycommandeverytimeadumpfileisrotated.Youcanreferencethedumpfileandthebasedirectorywiththe%d/%fmacros.Thismeansthatnfdumpwillalwaysbeexecutedonnewdataandonlyonceperdumpfile.Suddenly,youwon’tneedtoscheduleanycomplicatedcronjobtogeneratethefinal,human-readablenetflowlogfile.Wealsoaddeda-qoptiontosuppresstheheaderandstatisticsprintingtokeepthelogfileclean.
www.it-ebooks.info
NoteYoumightstillwanttoconfiguresomelogrotationforthe/var/nfdump/logs/zabbix_netflow.logfile.Ifyouletitgrowunchecked,itwillfillupyourdiskspaceinduetime!
It’sfinallytimetomakeZabbixawareofthenetflowlogfile.
MonitoringalogfilewithZabbixAsalreadyexplained,logfilemonitoringneedsaZabbixagent.Forillustrationpurposes,wewillassumethatyouhaveinstallednfdumponthesameboxastheZabbixserver,andthatthelogfileisthuslocallyavailable.Itgoeswithoutsayingthatyoucouldalsoinstallnfdump,alongwithaZabbixagent,onaseparated,possiblydedicatedmachine.Itwon’tmakeanydifferencefromZabbix’sperspective.
Basicitemcreationisfairlystraightforward,justpointtheitemkeytothecorrectfilepathandyou’regoodtogo.Pleasenote,inthefollowingexample,thetimestampparsingfield:
Thisisallyouneedforbasiclogfilemonitoring.Forfurtherexplorations,thelogkeyacceptsdifferentoptions,amongwhichthemostinterestingarethoserelatedtoregularexpressionfilteringandoutputsothatyoucanalsocreateadditionalitemsthatwillonlyextracttheexactinformationyouneed(forinstance,bytespersecondofaflow)anduseitasrawdata,justasyouwoulduseanyotherZabbixitem.Zabbix’sownofficialdocumentationisexcellentinthisrespect,soyouareencouragedtofindoutmoreathttps://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/log_items.
www.it-ebooks.info
Onthenfdumpside,therearemanymoreoptionsandfeaturesavailabletonfdump,we’vereallyonlyscratchedthesurfacetokeepthingssimple.Wedon’thavethespacetofullyexploreithere,butifyou’rewillingtospendsometimeexploringthetool,you’llfindthatnfdumpisnotonlycapableofpowerfultrafficfiltering,justastcpdumpis,butitcanalsocreatestatisticsandaggregateddataonvirtuallyeveryaspectofaflow,fromnetworkportstopacketsizes,andsoon.CombinethiswithZabbix’spowerfulexternalscriptitems,andyoucaneasilyseethatyoucansliceanddiceyourdata;however,ifyouwant,bringitintoZabbixforfurtherprocessing,graphing,andalarming.Really,theskyisthelimitwhenyoulearntocombinethesetoolstogether.
www.it-ebooks.info
SummaryInthischapter,youhavelearnedthedifferentpossibilitiesZabbixofferstotheenterprisingnetworkadministrator.
Youshouldnowbeabletochoose,design,andimplementallthemonitoringitemsyouneed,basedonthemethodsillustratedintheprecedingparagraphs:simplechecksthataremoreusefulandpowerfulthanthenameimplies;theall-powerfulSNMPprotocol,bothasgetvaluesandastraps;logfilesingeneral;andtheinfinitelyusefulnetflowprotocol
ThenextchapterwillbuildontheinformationexposedinthischapterandwillfocusmoreonservermonitoringandhowtoextractinformationfromDNSservers,webservers,proxies,andotherappliances.Theseareimportant,ifoftenoverlooked,componentsofanetworkevenfromtheperspectiveofanetworkadministrator,andyou’llfindmanyusefultipsonhowtomonitorthem.
www.it-ebooks.info
Chapter3.MonitoringYourNetworkServicesIneveryenvironment,especiallyinalargeone,therearemanynetworkcriticalservicesthataredirectlytiedonthenetworkinfrastructure.Manyofthemcanbemonitoredbythesystemadministrators,butthecorecriticalservicesforthewholenetworkarebetteriftheyaremonitoreddirectlybythenetworkadministrator.
Betweenthosecriticalservices,wecanfindthefollowing:
DNSDHCPNTPApacheproxy/reverseproxiesProxycacheSquid
Asitiseasytounderstand,evenifthoseservicesareprovidedfromsomededicatedserverandnotnetworkdevices,themetricsthatyouareacquiringfromthemarefundamental.Thosemetrics,indeed,playacriticalrolewhenyouwouldliketosetupaproactivealarm.
AnexampleofaservicethatcancausealotofconfusioninyournetworkcanbetheDNS,theDHCP,oreventheNTP.Inanidealenvironment,allthoseservicesneedtoberesponsive,andeventheresponsetimeiscrucial;ifeachoneofthosecomponentsbecomesunresponsive,theywillactastheweakestlinkofyourinfrastructure,causingalotofproblemsthatwillbequicklypropagatedtothewholenetwork.AsimpleNTPservercanintroduceconfusioninthelogsofyoursystemsorevencauseanissueinyourconnections.Workingonapracticalexample,trytoimaginethatyouhaveallyouraccountsstoredinanLDAP.Well,iftheLDAPtakestoomuchtimetoresolvetheUID/GIDofyouraccount,youcanhaveissuespropagatedtoallyoursystems.AnunresponsiveLDAPcancausefilesystemissuesandevenNASissues,andifallyouraccountsarestoredthere,evenanlscanliterallytakeages,withabigimpactonthewholeinfrastructure.Here,wearenotconsideringtheDNS,whereadysfunctioncanbeevenworse.
Also,thoseservicesneedtobetakenundersurveillanceas,iftheybecomeunresponsive,quitesoontheywillaccumulaterequeststoserve,andiftheenvironmentisnotready,theywillbefloodedbytheirownqueriesinaqueue,withaglobalimpactonourinfrastructure.
Inthischapter,wewillgothroughallthemainservicesthatanetworkadminshouldmonitortoavoidthesekindsofissues.Then,thereaderwilllearnandunderstandtheimportanceofaneffectiveproactivealarmtoavoidaquickescalationofissuesacrossthenetwork.
www.it-ebooks.info
MonitoringtheDNSThefirstnetworkcomponentwewillanalyzeandseehowtomonitoristheDNS.
ThemostpopularDNSserverisBIND,whichisalsooneoftheoldestpackagesproduced.Here,inthenextexample,weassumeyouhaveBIND9.6orlater.
Startingwithversion9.6,thereisabrandnewfeaturethatisnotevenmentionedinthemainpage(ofRedHatLinuxatleast).Thisfeatureisabuilt-inwebserverthatprovidesstatisticsaboutBINDinaverysimplewaythoughtHTTP.Toenablethisfeature,itisenoughtoaddthoselinestoyourBIND9configurationfile,/etc/named.conf:
statistics-channels{
inet127.0.0.1port8053allow{127.0.0.1;};
};
Thelinewehavejustaddedisagoodexampleasthestatistics’accessiscontrolledandrestrictedtothelocalhost.
TipBIND,bydefault,willusethestandard80HTTPportifyoudon’tspecifytheport.Alsopleasetakecaretolimittheaccesstothestatisticchannel;todoso,youcanusethisclause:
allow{address_match_list}
Ifyoudon’tspecifytheallowclause,BINDwillacceptconnectionsfromanyaddress.Thisneedstobeavoided.
Oncethisisdone,allyouhavetodoisrestartyourservicewith:
$servicenamedrestart
Stoppingnamed:[OK]
Startingnamed:[OK]
Now,youcanevenusecurltocallyourwebserverandhavedeliveredtoyouallthestatistics:
#curlhttp://127.0.0.1:8053
<?xmlversion="1.0"encoding="UTF-8"?>
<?xml-stylesheettype="text/xsl"href="/bind9.xsl"?>
<iscversion="1.0">
<bind>
<statisticsversion="2.2">
<views>
<view>
<name>_default</name>
<zones>
….
<summary>
<TotalUse>5965501</TotalUse>
<InUse>1502936</InUse>
<BlockSize>4718592</BlockSize>
www.it-ebooks.info
<ContextSize>3595936</ContextSize>
<Lost>0</Lost>
</summary>
</memory>
</statistics>
</bind>
</isc>
Now,wehavetwowaystoretrievethestatistics:
ConfigureBINDtowritethestatisticsinthestatfile(oldmethod)ConfigureBINDtousethebuilt-inHTTPwebservice
Thefirstandoldmethodcanbeusedforserversthatarenotunderaheavyload;thenewmethodusingthestatistics-channelsisontheotherhandlightweightandveryeasytomanage.Nowadaysthisoneisthepreferredmethodtouse.
NoteStartingfromBIND9.10,thestatisticscanbedeliveredineithertheXMLortheJSONformat.ThepreviousversionofBINDofferedonlystatisticsonXMLv2orV3.StartingwithBIND9.10,theXMLstatisticsareavailableonlyinV3format.Anyway,theJSONformatissignificantlyfasterthanXMLandevenlightweighttoprovide.
Now,tofiltertheoutputobtainedbycurl,thereisaninterestingutilitythatunfortunatelyisnotastandardRPMdistributedbyRedHat.Thetoolwearegoingtouseonthoseexamplesisxml2.
Thisxml2isanXMLprocessingtoolthatcanbeusedtoparseandreadtheXMLenvelopesandrewritethemasaflatformat.Theflatformatisreallyusefultobemanipulatedwithshellscripts.Then,firstofall,youneedtodownloadthisutility(thesourcecodeisavailableathttp://download.ofb.net/gale/xml2-0.5.tar.gz).Here’stheoutputsummary:
#wgethttp://download.ofb.net/gale/xml2-0.5.tar.gz
--2014-11-0110:43:44--http://download.ofb.net/gale/xml2-0.5.tar.gz
Resolvingdownload.ofb.net…64.13.131.34
Connectingtodownload.ofb.net|64.13.131.34|:80…connected.
HTTPrequestsent,awaitingresponse…200OK
Length:86318(84K)[application/x-gzip]
Savingto:"xml2-0.5.tar.gz"
100%[===================================>]86,318155K/sin0.5s
2014-11-0110:43:45(155KB/s)-"xml2-0.5.tar.gz"saved[86318/86318]
Performthefollowingstepstoobtaintheresultssetoutintheprecedingparagraph:
1. Explodethepackage,asfollows:
#tar-zxvfxml2-0.5.tar.gz
xml2-0.5/
xml2-0.5/configure.ac
xml2-0.5/aclocal.m4
…
www.it-ebooks.info
xml2-0.5/csv2.c
xml2-0.5/xml2.c
2. Stepintothedirectory,asfollows:
#cdxml2-0.5
3. Runtheusual./configurefollowedbymakeandmakeinstall,asfollows:
#./configure&&make
Then,asroot,youcannowrunthefollowingcommand:
#makeinstall
Onceallthishasbeencompleted,youarereadytoruntheutility.
Tomakeyoubetterunderstandwhatthistoolexactlydoes,youcanrunthefollowingcommand:
#curlhttp://localhost:8053/2>/dev/null|xml2|grep-A1queries
/isc/bind/statistics/server/queries-in/rdtype/name=A
/isc/bind/statistics/server/queries-in/rdtype/counter=11230
/isc/bind/statistics/server/queries-in/rdtype
/isc/bind/statistics/server/queries-in/rdtype/name=AAAA
/isc/bind/statistics/server/queries-in/rdtype/counter=1112
Now,theoutputisfinallyveryeasytomanipulatewithastandardutilitylikesedorawk.
4. Then,thenextsteptoenquirefromthelocallyinstalledagentistoaddthesetwolines:
UserParameter=bind.queries.in[*],curlhttp://localhost:8053/
2>/dev/null|/usr/local/bin/xml2|grep-A1
"/isc/bind/statistics/server/queries-in/rdtype/name=$1$"|tail-1|
cut-d=-f2
UserParameter=bind.queries.out[*],curlhttp://localhost:8053/
2>/dev/null|/usr/local/bin/xml2|grep-A1
"/isc/bind/statistics/views/view/rdtype/name=$1$"|tail-1|cut-d=-
f2
Usingtheprecedingcommandasanexample,youcanrunthestandardqueries,suchasA,AAAA,CNAME,ANY,MX,NS,PTR,SOA,andTXTrecordsin/out.
Now,ontheZabbixserverside,youneedtoconfigureallyouritemsjustastheoneshowninthescreenshotfollowingtheupcominglist,takingcaretocreatethesamekindofitemforAaswell:
AAAA
CNAME
ANY
MX
NS
PTR
www.it-ebooks.info
SOA
TXT
Onceyou’veaddedallyouritemsinagraph,thefinalresultwillbejustliketheoneshowninthenextscreenshot.Now,you’reacquiringallthequeriesdoneforthemostimportantDNSfields.
www.it-ebooks.info
DNS–responsetimeNow,wearemonitoringallqueriesdoneagainstallthemainDNSrecords,butactuallyweneedtocheckhowourDNSisworkingandthenhowmuchtimeisrequiredtohavetheresponseback.
OntheZabbixhow-to,thereisanexampletodowhat’savailablehere:https://www.zabbix.com/wiki/doku.php?id=howto/monitor/services/monitor_dns_and_ntp_services_on_your_network.
Theproblemwiththisexampleisthatthescriptandcodeproposedsimplyreturnsa0or1dependingontheDNSresponseorDNStimeout.
Well,thatexampleisnotgoodenoughforus;wearelookingfornumberslikeresponsetime,andoverthosenumberswecanimplementatrigger.ThetriggerneedstogoonfirewhenthetimeneededbyDNStogiveusbackaresponseishigherthanavaluethatwecanconsideracceptable.Inacomplexnetwork,youcanhaveaDNSquerywhereyoucantolerateaslowresponse(theentiredevelopmentnetworksegment,forinstance,isnotascriticalastheproductionsegment).Then,thesolutionsweproposeheregiveustheresponsetime.Wecanbuildourtriggerovertheresponsetimeunliketheotherway,whichisalotlessflexible.
Wecanseethescriptstepbystep;firstofall,weneedtoacquiretheresponsetime.Thiscanbedoneusingdig,asfollows:
#digmydomain.com
NoteNOTE
digispartofthebind-utilspackage.Ifyoudon’thaveitinstalledinyoursystem,youneedtorunasrootthefollowingcommand:
yuminstallbind-utils
Anyway,digusesthelocalresolver,andthenifyourunthesamequeryagain,you’llseethatthetimespenttoacquiretheDNSrecordis0minutes.Thisisclearlyafalsevalue!Toavoidanycachedresponseandtomeasuretherealtime,weneedtousethe+traceoption.Whentracingisenabled,digmakesiterativequeriestoresolvethename;practically,digwillfollowreferralsfromtherootservers,showingtheanswerfromeachserverthatwasusedtoresolvethelookup.
Here,weneedtohavethetotaltimespentforthequeryandnotthetimeconsumedbyeveryserver.Todothat,wecanusethefollowingsyntax:
$([email protected]+trace)
real0m1.376s
user0m0.010s
sys0m0.012s
Nowthatwehaveunderstoodthelogic,hereisthefullscriptwewilluse:
www.it-ebooks.info
#cattest_dns.sh
#!/bin/sh
iftest-z"$1";then
echo"YouneedtosupplyaDNSentrytocheck.Quitting"
exit01;
fi
DOMAIN=$1
MYTIME=$((timedig$DOMAIN+trace)2>&1|grepreal|awk-F'[m,s]''{print
$2}')
if[$?-eq0];then
echo$MYTIME
else
echo0
fi
Thisscriptrequiresa$1parameter,whichisthedomaintocheck.Now,weneedtoenablethisscriptontheagent’ssidewithUserParameterontheagentconfigurationfile,thusadding:
UserParameter=dns.responsetime[*],test_dns.sh$1
Thescriptwejustcreatedneedstoplacedinavalidruntimeagent’spath,orweneedtousethefullyqualifiedpathinUserParameter,asfollows:
UserParameter=dns.responsetime[*],/full/path/of/test_dns.sh$1
NoteThismethodisreallyusefulasyoucandeploythescriptondifferentnetworksegments,likeforinstance,theapplicationserverzone,andhavearealvalueofthetimeneededtoresolveaDNShostfromthatnetworksegment.
Asthelaststep,createtherelativeitemontheZabbixserverside,whereyouwillpasstheDNSnametocheck,asshowninthefollowingscreenshot:
Pleasebearinmindthatthisscript,ifexecutedcontinuously,canhammeryourDNSexactlybecauseitavoidsusingthecacheofthelocalresolverandevenoneoftheintermediatesegments.
Then,aswehaveexplained,weneedtoscheduleourscriptwithareasonableperiodthatcanbeforaninstanceof1minute.Pleaseconsideryournetworksegmentsfromwhichyou’rerunningthischeck,forboththequantityofscriptsthatarerunningandfrequency.
www.it-ebooks.info
NoteHere,youcancreateatriggerbasedonthezone,bearinginmindthatyou’remonitoringtheDNSresponsetimedirectlyfromthehoststhatrequirethoseDNSentriesresolved.Here,itisimportanttotuneyourtriggerbasedontheresponsetimeyouconsideracceptablefromthepointofviewofthezone.
Whenyou’recreatingyourtrigger,itisimportanttoconsiderthatthispluginprovidesyouwiththerealDNSresponsetime,whichistheworst-casescenario.Here,weavoidusinganycachingsystems,whichisnottherealcasebutapessimisticone.Thatsaid,ifyounoticesomespikesofhighresponsetime,thosecanbeignoredasthosespikescan’timpactyoursystem.Consideringthat,thetriggerneedstobetunedtospottheresponsetimethatisstilltherefortwoorthreeitemcycles(orevenmore—thisdependsonthefrequencyatwhichyourunthecheck)andavoidconsideringsinglespikes.
www.it-ebooks.info
DNSSEC–monitoringthezonerolloverHere,wedon’thaveenoughpagestoexplainallthefeaturesaddedbyDNSSECoracompletesetupguideofit.Anyway,itisimportanttoknowthatthebestwaytoavoidissueslikeaDNScachepoisoningattackistouseDNSSEC.DNSSECdoesadeepusageofcryptographickeysanddigitalsignaturestoensurethatlookupdataiscorrectandconnectionsarelegitimate.Then,inasecureenvironment,you’resupposedtousemainlyDNSSEC,andthenitisimportanttomonitorthecriticalDNSSECparameters;thoseitemscanberesumed,asfollows:
Thezonefile’svalidityThezones’rolloverstatusTheDNSresponsetime
Currently,therearetwopluginsavailabletoimplementchecksagainsttheDNSSECzonerollover:
RollstateZonestate
Thefirstonechecksthezonemanagedbythedaemonrollerd;thesecondonechecksthevalidityofDNSzones.
NoteThefullcodeisavailableathttps://github.com/hardaker/dnssec-tools/tree/master/dnssec-tools/apps/zabbix,andthepackageisavailableathttp://www.dnssec-tools.org/download/dnssec-tools-2.1.tar.gz.
OneoftherequirementstoproperlysetupthispluginisthatyouneedtobeawareofthefrequencyofyourrolloveractionstotunetheZabbixitem;pleasebeawarethatalittlelatencyisnormalhere.Anyway,aslongasyoudon’trolloverzoneseveryfewminutes(TTLissettoafewminutes),thislagwillnotbeanissue.
Now,beforeyoucanruntheplugin,youneedtohaveinstalledafewrequiredPerlmodules:
#perl-MCPAN-eshell
cpan>installNet::DNS
cpan>installNet::DNS::SEC
Wearesupposingthatyoualreadyhavecpaninstalled;ifyoudon’thaveitinstalledinyoursystem,pleaseinstallitwiththefollowinglineofcode:
#yuminstallcpan
Now,onceyouhaveinstalledtherequiredmodule,youneedtoinstalltheopenssl-develpackagewiththefollowingcommand:
#yuminstallopenssl-devel.x86_64
Now,youcanfinallyuncompressthesoftwarewiththefollowingcode:
www.it-ebooks.info
#tar-zxvf./dnssec-tools-2.1.tar.gz
#cd./dnssec-tools-2.1
#./configure&&make&&makeinstall
Nowin/dnssec-tools-2.1/apps/zabbix/,wehavealltheneededsoftware.Herearethepiecesofsoftwareavailablein/dnssec-tools-2.1/apps/zabbix/:
#ls-l
total40
-rwxrwxr-x.112741274768Jan22013backup-zabbix
-rw-rw-r--.1127412741706Jan22013item.fields
-rw-rw-r--.1127412742878Jan22013README
-rwxrwxr-x.1127412746763Feb152013rollstate
-rwxrwxr-x.1127412747720Feb152013uemstats
-rw-rw-r--.1127412741329Oct192011zabbix_agentd.conf
-rwxrwxr-x.1127412746314Feb152013zonestate
Finally,wecantryournewplugins,asfollows:
#./rollstatemydomain.com
ZSKphase3
#./zonestatemydomain.com
zonefilevalid
Now,it’stimetoenableournewplugins;todothis,weneedtodefineacoupleofnewentriesofUserParameterontheagentside’s/etc/zabbix/zabbix_agentd.conf:
UserParameter=dnssec-tools.rollover.status[*],rollstate$1
UserParameter=dnssec-tools.rollover.statusnum[*],rollstate–numeric$1
Evenhere,youneedtoplacetherollstateplugininadirectorycontainedinthepathorusethefullyqualifiedpathforourplugin.Also,onceyouhaveaddedUserParameter,youneedtorestarttheagentwith:
#servicezabbix-agentrestart
ShuttingdownZabbixagent:[OK]
StartingZabbixagent:[OK]
Therollstatepluginprovidestwodifferentoutputswiththe–numericoptionspecified.ItprovidespositivenumbersfortheZSKphasesandnegativenumbersfortheKSKphases.ThisenablesustoproduceagraphthatrepresentsallthephasesofDNSSEC.
OnceyouhavecreatedtheZabbixagentitemonyourtemplateandyourscriptisrunning,theoutputwillbelikethenextscreenshot.
Intheexampleandtherelativegraph,wehaveahighlyfrequentrollover.Inareal-lifescenario,thetimerequiredtogothroughallthedifferentstatuseswillbelonger.
www.it-ebooks.info
ThedetailsoftheDNSSECrolloverintextmode,usefultokeeptrackofallthestatuschanges,willbecontainedinatextitem.Anexampleofthelatestdataisshowninthenextscreenshot:
Asyoucansee,youwillhaveahistoricalstatusofallthestepscrossedduringtherollover,andyouwillhaveacleartrackofthestepsperformed.
NoteThisitemwillbepreciousifyourprocessgetsstuckonastep,especiallyifthishappensperiodically.
Inthenextscreenshot,youcanseethezonestatuspluginatwork:
www.it-ebooks.info
Now,theonlythingyoustillhavetodoiscreateatriggerbasedontheinformationwe’reacquiring.Here,itisimportanttobearinmindthatalittlelagisnormalduringthezonetransferprocess;thislagneedstobeconsideredwhenyousetupthetrigger.
www.it-ebooks.info
ApachemonitoringMostofthereverseproxiesarenowadaysimplementedusingApache.Apache,otherthanbeingawebserver,isquiteusefulasareverseproxyasitincludessomepowerfulmodules:
mod_proxy
mod_proxy_http
mod_proxy_ftp
Otherthanasareverseproxy,itcanbeusedasaloadbalancerthanksto:
mod_proxy_balancer
Now,unfortunately,thereisn’tavalidmethodtoacquirethemetricsstrictlyrelatedtothemoduleused,butanyway,wecanacquirequiteafewmetricsfromApacheitself.
Thefirstthingyouhavetodobeforeyoucanacquirethestatisticsisenablethem.Todothis,youneedtoputthefollowinglinesinyourApacheconfigurationfile:
<Location/server-status>
SetHandlerserver-status
Allowfrom127.0.0.1
Orderdeny,allow
Denyfromall
</Location>
Also,youcanoptionallyaddthefollowinglinetoyourglobalApacheconfigurationfile:
ExtendedStatusOn
Here,weareconfiguringthemodulewiththeExtendedStatusOnoption.Withthissetting,Apachekeepstrackofextendedstatusinformationforeachrequest.Thiscollectioncanslowdowntheserver,andifyounoticeperformanceissues,itcanbedisabledwiththeExtendedStatusOffkeyword.
TipPleasekeeprestricted,asmuchasyoucan,theaccesstothe/server-statuslocation.Inourcase,itisallowedonlyfrom127.0.0.1.ThismeansthatyouneedtocollectthestatisticsfromtheagentinstalledlocallyonyourApachehost.Itisimportanttoknowthatifmod_statusiscompiledintotheserver,thenitshandlerisavailableinallconfigurationfiles,includingper-directoryfiles,likehtaccess.Thiscanhavesecurity-relatedramificationsforyoursite.
Now,allyouhavetodoisrestartyourApacheandcheckwhetheryoucanretrievethestatisticsrunningthefollowingcommand:
[root@localhost~]#curlhttp://127.0.0.1/server-status
<!DOCTYPEHTMLPUBLIC"-//W3C//DTDHTML3.2Final//EN">
<html><head>
<title>ApacheStatus</title>
</head><body>
www.it-ebooks.info
<h1>ApacheServerStatusfor127.0.0.1</h1>
<dl><dt>ServerVersion:Apache/2.2.15(Unix)DAV/2PHP/5.3.3</dt>
<dt>ServerBuilt:Jul23201414:17:29
</dt></dl><hr/><dl>
<dt>CurrentTime:Monday,03-Nov-201419:48:11PST</dt>
<dt>RestartTime:Monday,03-Nov-201419:48:00PST</dt>
<dt>ParentServerGeneration:0</dt>
<dt>Serveruptime:11seconds</dt>
<dt>Totalaccesses:9-TotalTraffic:0kB</dt>
ThisApachemodule’soutputisreallyfullofusefulinformation;lookingattheoutputindetail,youcanseethatitprovidestheinformationshowninthefollowingscreenshot:
Here,youhaveaviewthatissplitintofourmainsections,whichareasfollows:
TheApacheversiondata,modulestarted,andserverbuilddetailsTheApacheserverstatusthatprovidesyoutheuptime,CPU,numberofaccess,numberofrequest/sec,andsomemoreinformationaboutitsstatusTheApachescoreboardAsectionwithallthedetailsoftheconnectionserved
Here,retrievingthestatisticsisnotaseasyasyouwouldimagine.Thefirstandsecondsectionsarequiteverbose,anditiseasytoextracttherequiredinformationfromthemonceyou’veobtainedthewebpage.ThethirdsectionisalittlemorecomplexasitistheApachescoreboard.ThescoreboardisarepresentationofApache’sworkersandtheirrelativestatus.TheworkersareApache’srequest-handlerstatus.Thekeysusedonthescoreboardarethefollowing:
www.it-ebooks.info
ScoreboardKey:"_"WaitingforConnection,"S"Startingup,"R"Reading
Request,"W"SendingReply,"K"Keepalive(read),"D"DNSLookup,"C"Closing
connection,"L"Logging,"G"Gracefullyfinishing,"I"Idlecleanupof
worker,"."Openslotwithnocurrentprocess
Then,toretrieveandanalyzethestatus,weneedtouseaslightlydifferentURL:http://localhost/server-status?auto.
WecantrytheoutputproducedbythisURLusingcurl,asfollows:
#curlhttp://127.0.0.1/server-status?auto
TotalAccesses:1334
TotalkBytes:2163
CPULoad:5.20713
Uptime:2776
ReqPerSec:.480548
BytesPerSec:797.879
BytesPerReq:1660.35
BusyWorkers:1
IdleWorkers:10
Scoreboard:
_______W___…...............................................................
...........................................................................
...........................................................................
.............................
Now,it’seasytoretrievetheCPULoadvalue,forinstance:
#curl-shttp://127.0.0.1/server-status?auto|awk'/^CPULoad:/{print
$2}'
5.15882
Withthesamemethod,wecanacquireallthemetrics,forexample,thenumberofIdleWorkerswillbe:
#curl-shttp://127.0.0.1/server-status?auto|awk'/^IdleWorkers:/
{print$2}'
10
Parsingthescoreboardisalittledifferentasweneedtocountthenumberof_ifwearelookingatalltheworkersthatarewaitingforaconnectioninsteadofcountingalltheoccurrencesofWtocheckalltheworkersthataresendingreplies.Toaddressthisrequirement,youcanusethefollowingcommand:
#curl-shttp://127.0.0.1/server-status?auto|awk'/^Scoreboard:/
{print$2}'|awk'BEGIN{FS="_"};{printNF-1}'
10
ThefirstawkcommandidentifiestheScoreboard:section,thesecondawkcommandcountsalltheoccurrencesof_intheline,definingafieldseparator,andthencountingallthematchedfields.
Currently,therearethreeprebuiltpluginstodothis:
zapache:ThisisashellscriptcalledviaUserParameterZabbixApacheUpdater:ThisisaPythonsoftwarethatneedstobescheduledon
www.it-ebooks.info
crontabquery_apachestats.py:ThisisaPythonsoftwaretriggeredbyUserParameter
Inthissection,wewillanalyzezapacheasitusesthesamemethoddescribedtoacquiremetricsfrommod_statusofApache.Thescriptisavailablefordownloadathttps://github.com/lorf/zapache.
Allyouhavetodoisdownloadzapachefromthatlocation,copyzapacheunder/home/zabbix/bin/withtherelativetemplate,andthenconfigureUserParameterintheagentconfigurationfile/etc/zabbix/zabbix_agentd.conf,asshownhere:
UserParameter=zapache[*],/home/zabbix/bin/zapache$1
Now,ontheGUI,youhavetocreateyourtemplateorimporttheonedistributedwithzapache.Then,navigatetoConfiguration|Template|Importandselectthezapache-template.xmltemplateifyouwanttheitemasZabbixagentorthezapache-template-active.xmltemplateifyouprefertheitemsmanagedasZabbixagent(active).
Ifyoutakealookatthezapachesourcecode,youwillnoticethatitcanrunasZabbixagent’smodeorasanexternalscript,whichmeansthatyoucanuseittoacquiretheApachestatisticslocallyonthesameserverorremotely.
Hereisthecodesectionthatmanagesthiskindofbehavior:
if[[$#==1]];then
#AgentMode
STATUS_URL="http://127.0.0.1/server-status?auto"
CASE_VALUE="$1"
elif[[$#==2]];then
#ExternalScriptMode
STATUS_URL="$1"
case"$STATUS_URL"in
http://*|https://*);;
*)STATUS_URL="http://$STATUS_URL/server-status?auto";;
esac
CASE_VALUE="$2"
Asyoucansee,youcanrunthescriptwithonlyoneparameter,whichrepresentsthemetricyouwouldliketoacquire,ortwoparameters,specifyingeventheremoteIPaddressofyourApachereverseproxyorwebserver.Here,inordertokeepthingseasy,weavoidmod_statusfrombeingaccessedexternallyusingaUserParameter.Anyway,itisbettertobeawarethatyoucanevencentralizestatisticacquisitionthankstothiscodesection.
ThefinalresultofoursetupandApache’smetricacquisitionisshowninthenextscreenshot:
www.it-ebooks.info
Now,itistimetodiscusstriggersrelatedtothisApachemonitoring.Firstofall,youneedtocreateatriggerbasedonthelastvalueofzapacheping,asfollows:
{TemplateAppApacheWebServerzapache:zapache[ping].last(0)}=0
Ofcourse,ifthezapachepingfails,returning0,youhaveanissue.Someotherparametersthatarecriticalforserverstatusandonwhichyoucancreatetriggersare:
WaitingForConnection:ThisindicatesthatthenumberofprocessesarewaitingforaconnectionReqPerSec:ThisindicatesthenumberofrequestspersecondCPULoad:ThisindicatestheamountofCPUconsumedbyApache
Thosevaluesarestrictlydependentontheserveryou’reusing,thenumberofclientsyouareserving,andmostimportantly,whatexactlyandhowyouareservingtherequest.Aboutwhatandhowyouareservingtherequest,youcanhavesomeverycomplexrewritingandreverserulesthatcanmakeagroupofURLsmorecomplextomanage.Here,thebestthingtodoistrytofindoutyourApache’slimitusingsometoolsthatareabletoproducealotofconcurrentconnectionsandthenworkload,forinstance,youcantrySiege.
NoteMoreinformationaboutSiegeisavailablehere:http://www.joedog.org/siege-home/.
Onceyou’vetestedandfoundthemaximumnumberofclientsyoucanserveperURLandyou’veseenthewebserverlimits,youcancreateandtuneyourcustomtriggers.
www.it-ebooks.info
NTPmonitoringThesystemclockissomethingyoushouldkeepmonitoringbecauseif,forsomereason,yoursystemsuffersasystemclockdrift,thiscanbecomeabigissue.
Performingapracticalexampleofheavydriftonthesystemclockwillcauseissues.TheDNSSECzonereplication,yourFTPservice,theIMAPservice,andmanyotherserviceswillbeaffected,makingyourserverunstableandunusable.
TokeepyoursystemclockinsyncwiththeremoteNTP,youcanuseandinstalltheNTPdaemonthatwilltakecareofthesystemclock.
ToinstallNTP,youcanuseyumasusual:
#yuminstallntp
...outputremovedhere…
Installed:
ntp.x86_640:4.2.6p5-1.el6
Complete!
Onceyou’veinstalledtheNTP,youneedtofindtheserverthatisclosertoyouusingthewebsitehttp://www.pool.ntp.org/en/.
Fromthiswebsite,youneedtochoosetheserverthatisbetterforyouandthenchangethe/etc/ntp.confconfigurationfile.
Also,itisagoodpracticetoaddthelogfiledirectiveattheendofthentp.confconfigurationfile,asfollows:
#echo"logfile/var/log/ntp.log">>/etc/ntp.conf
Thenstartorrestarttheservice,asfollows:
#servicentpdstop
Shuttingdownntpd:[OK]
#servicentpdstart
Startingntpd:[OK]
Now,youneedtoconsiderthatyoucanhaveonecentralserverusedasaprimaryntpdserverforyournetworkandpropagatethesystemtimefromthere;inthiscase,youneedtochangethe/etc/ntp.confconfigurationfileabit:
#Hostsonlocalnetworkarelessrestricted.
restrict192.168.1.0mask255.255.255.0nomodifynotrap
Nowfinally,youcanattachallthehostsofyournetworktothisntpdserverandthenmonitorthisNTPandtheclient’stime.
TipIfyouareprotectingaserverwithafirewall,youneedtoenabletheUDPonport123onbothdirections.Ifyou’reusingiptablestoenabletheclientandtheservercommunication,youneedtoaddthefollowingrulestotheOUTPUTandINPUTchains:
www.it-ebooks.info
iptables-AINPUT-pudp--dport123-jACCEPT
iptables-AOUTPUT-pudp--sport123-jACCEPT
Now,toretrievemetrics,weneedtoqueryntpd.Forthisoperation,wecanusentpq,whichwillshowallthestatistics.Fromamonitoringperspective,we’relookingfortheoffset,jitter,anddelay.
Inthenextexample,weseethecompleteoutputofntpq,asfollows:
#ntpq-pn127.0.0.1
Remoterefidsttwhenpollreachdelayoffsetjitter
==================================================================
+91.247.253.152191.241.139.1373u964135.27629.4929.791
+217.147.208.1194.242.34.1492u864119.61730.91211.497
*192.33.214.47129.194.21.1952u764125.58132.15711.007
+195.141.190.190212.161.179.1382u664120.73931.14310.983
Pleasenotethatthisserverissufferingabigdriftandthetriggerisalreadyonfire.
Toacquirethemetricthen,wecanuseacommandlikethisone:
#ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{offset=0}$1~/\*/{
offset=$9}END{printoffset}'
32.157
ThiscommandretrievestheoffsetbetweenthesystemclockandtheNTPserver.
NoteWeareusingthe–pand–noptionstogether;withthe–noption,weareavoidingthenameresolution,andthentheDNSquery.Thisisdoneinordertokeeptheitemaslightweightaswecan.
Now,wecanquicklysetupNTPmonitoringusingUserParameterontheagentsidewith:
UserParameter=ntp.jitter,ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{
offset=0}$1~/\*/{offset=$9}END{printoffset}'
ThiswillsetUserParametertoretrievethejittervalue;anyway,wecanevendosomethingalittlemorecomplexandthenproduceascriptlikethefollowing:
#!/bin/bash
VERSION="1.0"
functionusage()
{
echo"ntpcheckversion:$VERSION"
echo"usage:"
echo"$0jitter-Checkntpjitterdelay"
echo"$0offset-Checkntpoffset"
echo"$0delay-Checkntpdelay"
}
########
#Main#
########
if[[$#!=1]];then
#NoParameter
usage
www.it-ebooks.info
exit0
fi
case"$1"in
'jitter')
value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{jitter=0}$1
~/\*/{jitter=$10}END{printjitter}''"
rval=$?;;
'offset')
value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{offset=0}$1
~/\*/{offset=$9}END{printoffset}''"
rval=$?;;
'delay')
value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{delay=0}$1
~/\*/{delay=$8}END{printdelay}''"
rval=$?;;
*)
usage
exit1;;
esac
if["$rval"-eq0-a-z"$value"];then
rval=1
fi
if["$rval"-ne0];then
echo"ZBX_NOTSUPPORTED"
fi
echo$value
Then,ontheagentside,wecandeploythisscriptcalledntpcheck.shinthe/home/zabbix/bindirectory:
#ls-la/home/zabbix/bin/ntpcheck.sh
-rwxr-xr-x1zabbixzabbix781Nov903:23/home/zabbix/bin/ntpcheck.sh
Oncethisisdone,allwehavetodoiscreateUserParameter,asfollows:
UserParameter=ntp[*],/home/zabbix/bin/ntpcheck.sh$1
Then,restarttheagent:
#servicezabbix-agentrestart
ShuttingdownZabbixagent:[OK]
StartingZabbixagent:[OK]
Testournewitems:
#zabbix_get-s127.0.0.1-kntp[jitter]
2.273
#zabbix_get-s127.0.0.1-kntp[offset]
-6.696
#zabbix_get-s127.0.0.1-kntp[delay]
18.956
Andintheend,createourthreenewitemsontheZabbixGUI,asshowninthefollowingscreenshot:
www.it-ebooks.info
NTP–whatarewemonitoring?Now,evenifthoseitemnamesappearassomethingeasytounderstand,itisbettertoknowwhatwearemonitoring.Firstofall,weneedtoclarifythatwe’reacquiringvaluesforthecurrenttimesource,hencewearetakingthevaluesinthelinethatbeginswitha*fromthentpqoutput.Forconvenience,thentpqoutputisreportedhere:
#ntpq-pn127.0.0.1
Remoterefidsttwhenpollreachdelayoffsetjitter
==================================================================
+91.247.253.152191.241.139.1373u964135.27629.4929.791
+217.147.208.1194.242.34.1492u864119.61730.91211.497
*192.33.214.47129.194.21.1952u764125.58132.15711.007
+195.141.190.190212.161.179.1382u664120.73931.14310.983
Asyoucansee,thelinesofthisoutputarenotordered,andtheybeginwith+and*(inthisexample).Weareinterestedintheonethatbeginswith*.Thereasonisthatthelinethatbeginswith*representsthepreferredandcurrenttimesource.
Wecanevenhaveaprefixlikethefollowing:
+:Thissignindicatesthatthepeerisagood,preferredremotepeerorserver(space),x,-,#,and.:Theseindicatethatthispeerisnotbeingusedforsynchronization
Now,wehaveclarifiedthereasonwhywearerunningthisawkcommand:
#ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{delay=0}$1~/\*/{delay=$8
}END{printdelay}'
Now,tohavesomemoredetailsaboutwhatwe’reacquiring,wecandefinethemas:
Delay:Thisisthecurrentestimateddelay.Itisthetransittimebetweenremotepeersorserversinmilliseconds.Offset:Thisisthecurrentestimatedoffset.Itisthetimedifferencebetweenremotepeersinmilliseconds.Jitter:Thisisthecurrentestimateddispersion,orbetter,thevariationindelaybetweenthesepeersinmilliseconds.
NoteIfyou’remonitoringaserverthatisrunninginavirtualenvironment,youneedtobeawarethatpracticallyallthevirtualizationsoftwaresuffersfromsystemclockdrift.Thencheckthevendor-specificbestpracticetoreducetheNTPdrift.
Nowit’stimetochangethescriptalittleaswecanchecktheNTPhealthstatusbyaddingthefollowingcasestatement:
case"$1"in
…
'health')
primary="'ntpq-pn127.0.01|grep^\*|grep-vgrep|wc-l'"
rval=$?
www.it-ebooks.info
if["${primary}"-eq"1"];then
value="1"
else
value="0"
fi
;;
…
esac
Now,wecancheckwhetherwehaveatleastoneprimarypreferredsourcedefinedtogettheNTPsyncinagoodshape.Weneedtothenaddanewitemandarelatedtriggerthatwillgoonfireifthevaluereturnedis0.Otherthanthistrigger,wecanevenhaveatriggerthatwillgoonfireiftheclockdriftisbiggerthan50millisecondsforinstance,orevenless.
Inthenextscreenshot,youseetheinteractionbetweentheJitter,Offset,andDelayonaLinuxvirtualserver(thatsufferfrombigsystemclockdrifts):
www.it-ebooks.info
SquidmonitoringSquidisthemostdiffusedcachingproxyfortheWeb.SquidsupportsHTTP,HTTPS,FTP,andmanymoreprotocols.Thisproxysoftwarereducesalotofthebandwidthrequiredtoserveitsclientsandimprovestheresponsetime,implementingaverygoodcachingsystem.Forallthosereasons,itisquiteevidentwhyyoushouldhaveSquidtomonitorinsideyournetwork.
TherearetwoprimarywaystoacquiredataandmetricsfromSquid:
UsingSNMPUsingsquidclient
Ifyou’recuriousabouttheSNMPsetupontheSquidserver,youcanhavealookattheofficialdocumentation,inparticularthesectionavailableathttp://wiki.squid-cache.org/Features/Snmp.
WeshouldavoidenablingSNMPonourSquidasithasbeenaffectedinthepastbymanyoverflowsandissues.Thelastsecurityissue,atthetimeofwritingthis,causedbySNMPenabledonSquid,isavailableathttp://www.squid-cache.org/Advisories/SQUID-2014_3.txt,andasyoucansee,itisareallyrecentissue.
Fortunately,theclientisreallypowerfulandthispermitsustoimplementagoodmonitoringsolutionwithoutenablingSNMP.
Typethefollowingcommand:
#squidclientmgr:info
Inresponsetotheprecedingcommand,Squidwillprintouttheentirestatisticdomainacquireduntilnow:
HTTP/1.0200OK
Server:squid/3.1.10
Mime-Version:1.0
Date:Sun,09Nov201417:23:25GMT
Content-Type:text/plain
Expires:Sun,09Nov201417:23:25GMT
Last-Modified:Sun,09Nov201417:23:25GMT
X-Cache:MISSfromlocalhost.localdomain
X-Cache-Lookup:MISSfromlocalhost.localdomain:3128
Via:1.0localhost.localdomain(squid/3.1.10)
Connection:close
...
Then,asyoucanunderstand,itwillbequiteeasytoretrievesomeimportantitemsfromthiskindofoutput.Tryingoutanexample,ifyouwouldliketoacquiretheCPUUsage,youcansimplyrun:
#squidclientmgr:info|grep'CPUUsage:'
CPUUsage:0.01%
Ofcourse,thiskindofoutputneedstobealittleshapedtobeusableforourwork,thenext
www.it-ebooks.info
commandwillbeaUserParameterreadycommand:
#squidclientmgr:info|grep'CPUUsage:'|cut-d':'-f2|tr-d'%'|tr-d'
\t'
0.01
Now,wehavetwowaysofdoingthis:
WecreatealonglistofUserParameterontheagentsideWecreatejustaone-userUserParameterandcallitusingaparameter
Thesecondwayisthepreferredapproachasifyouneedtoaddanitemtoacquire,youdon’tneedtorestarttheagent.Hereduetospaceconstraints,wewillnotcommentallthescript;forthecompletescript,pleaserefertoAppendixB,CollectingSquidMetrics.
YouneedtocreateUserParameter:
UserParameter=squid[*],/home/zabbix/bin/squidcheck.sh$1
Now,youneedtorestarttheagent,andyoucancheckwhetheryou’reabletoacquirethemetricswiththefollowingcommand:
#zabbix_get-s127.0.0.1-ksquid[icp_sent]
12
Ifyoucanretrievethemetrics,theconfigurationisfine.
Now,ontheserverside,youneedtocreateyouritems,asshowninthefollowingscreenshot:
Nowthatwearefinallyacquiringallthemetrics,itisimportanttodefineatleasttwo
www.it-ebooks.info
triggers:
OnetiedtothenumberofSquidprocessesrunningthatshouldneverbe0Onetiedtothenumberofavailablefiledescriptors;ifthisnumberislessthan100,weneedtohaveatriggeronfire
Thisisshowninthefollowingscreenshotandistheminimumnumberoftriggersyoushouldhave:
ToclosetheSquidmonitoring,wecantellthatyouarenowabletoacquireatleast22itemsusingthescriptavailableonGitHubathttps://github.com/smartmarmot/zabbix_network_monitoring/tree/master/Chapter3;youcannowsetmanyothertriggersdependingonyoursetup,servercapacity,numberofclientstoserve,andthemeanofthenumberofpagesrequiredbyyourclientnetwork.
Amongthemostimportantparameterstomonitor,wehave:
Thebytehitratioover5and60minutesTherequestdiskhitratioover5and60minutesRequestfailureratio
Allthehitratiosneedtobeascloseto100percentaspossible.Everyvalueofcachingunder70percentshouldmakeatriggergoonfire,andeventherequestfailureratio,ifitishigherthan30,shouldtriggeranalarmasitistellingusthatoursystemisnotrespondingproperly.
www.it-ebooks.info
SummaryInthischapter,wecoveredalargenumberofcomponents.Westartedourdiscussionfromthemostusedandevenverycriticalnetworkservice:DNS.Goingaheadonthesameway,wediscussedDNSSEC;then,wemovedontoApache,themostusedandeffectivereverseproxy;walkedthroughNTP;andclosedthechapterwithSquid,themostinstalledandusedproxyservice.Forallthesystemsandservicesanalyzed,you’renowabletoacquirethemostcriticalmetrics,andyouknowhowtocreateeffectivetriggers.
Triggersherearecoveringthemostcriticalroleandhenceyourexperiencewithinyournetworkisthetrulyaddedvalue.You,withtheknowledgeacquiredfromthischapterandyourenvironmentexperience,willbethekeytocreatingeffectiveandproactivetriggers.Thischapterhascoveredallthecriticalservicesyoucanfindinanetwork,andnowyoucaneasilyprovideaheavyaddedvalue,creatingproactivechecksandinstallinganeffective,tailor-mademonitoringsolution.Inthenextchapter,youwilllearnhowtoautomatethediscoveringofyournetwork’selementsandhowtoapplyatemplatetothediscovereditem.Also,youhavetoadaptyourmonitoringsystemwithinyourenvironments,andthiskindoftaskisthetypicalboringandtime-consumingtaskthatanetworkadmindoesn’tliketodo.Thechapterwillprovideyouwithallthenecessaryinformationtousethehostdiscoveryandthelow-leveldiscoveryinaneffectiveway.Youwillbeguidedthroughthedifficultwaytoautomatetheitemdiscovery:thiswillheavilyreducethetimeneededtostartupyourmonitoringsolutionbutwillimpactandreducethetimeneededtomaintainyourgrowinganddynamicallymovingsetup.
www.it-ebooks.info
Chapter4.DiscoveringYourNetworkInthepreviouschapters,we’veseenhowtogetdifferentmetricsfromquiteafewdifferentsources,usingdifferentmethods.Whatwehaven’tcoveredyet,ishowtoeasilygetallthisdataintoZabbixwhenyouhaveagreatnumberofmonitoredobjects.
Manuallycreatinghosts,items,andtriggersisanexcellentexercisetogetthehangofhowthingsworkinZabbix,butitcanquicklybecomearepetitive,boring,error-proneactivity.Inotherwords,theyarethekindsoftaskscomputersweremadeforinthefirstplace.
Whatifyourmonitoringsolutioncouldjustfindthehostsanddevicesyouwanttomonitor,addthemasZabbixhosts,applyatemplate,andstartmonitoringthem?Andwhatifitdidn’tjustlimititselftofindinghoststomonitor,butitalsofoundoutwhetheryourswitchhas24or48ports,howmanydisksyourwebserverhasattached,andwhatportsareopenonacertainhost?Aftersomeinitialconfiguration,youwouldnothavetobotherwithaddingorremovingthingstomonitor.Itwouldcertainlybegreat,buttheproblemwithautomateddiscoveryisthatitoftenhastocometotermswiththerealityofareal-worldnetwork,whichisoftenfullofexceptionsandspecialrules.Insuchcases,youcouldfindyourselfspendingalotoftimetryingtoadaptyourmonitoringsystemtoyourenvironmentinordertocatchupwithanautomateddiscoverythatmightbejustalittletooautomatic.
Luckily,Zabbixcansupportmanydifferentdiscoverystrategies,mixthemupwithregularhostanditemcreation,andgenerallyprovideagoodbalancebetweentheneedtohaveafullyautomatedsystemandtheneeddohaveamonitoringsolutionthatmatchesascloselyaspossibletheenvironmentithastomonitor,withallitsexceptionsandspecialcasesthatareimpossibletocapturewithjustadiscoverystrategy.
ThischapterwillbedividedintotwomainpartsthatmirrorthetwomainlevelsofdiscoverythatZabbixsupports:networkdiscoveryandlow-leveldiscovery.Theformerisusedtofindoutwhichhostsareinyournetwork,andthelatterisusedtofindoutwhatfacilitiesandcomponentsarefeaturedinagivenhost.
Let’sstartwithfindingouthownetworkdiscoveryworksandhowtomakethemostoutofit.
www.it-ebooks.info
FindinghoststheZabbixwayZabbix’sdiscoveryfacilitiesconsistofasetofrulesthatperiodicallyscanthenetwork,lookingfornewhosts,ordisappearingones,accordingtopredeterminedconditions.
ThethreemethodsZabbixcanusetocheckfornewordisappearedhosts,givenanIPrange,are:
TheavailabilityofaZabbixagentTheavailabilityofanSNMPagentTheresponsetosimpleexternalchecks(FTP,SSH,andsoon)
Thesecheckscanalsobecombined,asillustratedinthefollowingexample:
Asyoucansee,whenenabled,thisrulewillcheckeveryhour,intheIPrange192.168.1.1-254,foranyserverthat:
ReturnsanSNMPv3valuefortheSNMPv2-MIB::sysDescr.0OIDIslisteningtoandacceptingconnectionsviaSSHHasanHTTPSserverlisteningonport8000
Beawarethatadiscoveryeventwillbegeneratedifanyoneoftheseconditionsismet.
www.it-ebooks.info
So,ifadiscoveryrulehasthreechecksdefinedandahostinthenetworkrespondstoallthreechecks,threeeventswillbegenerated,oneperservice.
AsusualwithallthingsZabbix,adiscoveryrulewillnotdoanythingbyitself,exceptgenerateadiscoveryevent.ItwillthenbethejobofZabbix’sactionsfacilitytodetecttheaforesaideventanddecidewhetherandhowtoactonit.
Discoveryeventactionsareverysimilartoregulartriggereventactions,soyou’llprobablybealreadyabletomakethemostoutofthem.ThemainthingtorememberisthatwithZabbix,youcannotactdirectlyonaneventtocreateordisableahost:youneedtoeithercopytheeventdatabyhandsomewhereandthenproceedwithallthemanualoperationsneededbasedonthatdata,oryouneedtoproperlyconfiguresomeactionstodothatworkforyou.Inotherwords,withoutaproperlyconfiguredaction,adiscoveryrulewillnotaddbyitselfanydiscoveredhosttothelistofmonitoredones.
Everyactionhasaglobalscope:it’snottiedtoanyparticulartrigger,host,orhostgroupbydefault.Thismeansthatwhenyoucreateanaction,you’llneedtoprovidesomeactionconditionsinordertomakeitvalidonlyforcertaineventsandnotothers.ToaccessthediscoveryactionssectioninthewebUI,headtoConfiguration|ActionsandthenselectDiscoveryfromtheEventsourcedrop-downmenu,justundertheCreateactionbutton.
Whenyoucreateanaction,you’llstartwithgivingitanameanddefiningadefaultmessageintheactiondefinitionsection.You’llthenmovetotheactionconditionssectiontoprovidefilteringintelligence,beforefinishingwiththeactionoperationssectiontoprovidetheaction’scorefunctionality.Actiondefinitionsareprettysimpleasyou’lljustneedtoprovideauniquenamefortheactionandadefaultmessage,ifyouneedone.So,let’smovestraighttotheinterestingsectionsofactionconfiguration:conditionsandoperations.
www.it-ebooks.info
DefiningactionconditionsTheactionconditionssectionletsyoudefineconditionsbasedontheevent’sreportedhostIPaddress,servicestatusandreportedvalue,discoveryrules,andafewothers:
TheReceivedvalueconditionisofparticularinterest,asitallowsyoutodothingslikedifferentiatingbetweenoperatingsystems,applicationversions,andanyotherinformationyoucouldgetfromaZabbixorSNMPagentquery.Thiswillbeinvaluablewhendefiningactionoperations,asyou’llseeinthenextparagraph.Areceivedvaluedependsonthediscoveryruleandontheoutputofthediscoveryeventthattriggerstheaction.Forexample,ifadiscoveryruleissettolookforhostsrespondingtoanSNMPGetfortheSNMPv2-MIB::sysDescr.0OID,andthatrulefindsarouterthathasC3745asthevalueofthatOID,thenthediscoveryeventwillpassC3745totheactionasthereceivedvalue.
Singleconditionscanbecombinedtogetherwithlogicaloperators.There’snotmuchflexibilityinhowyoucancombinethemthough.
YoucaneitherhaveallAND,allOR,oracombinationofthetwowhereconditionsofdifferenttypesarecombinedwithAND,whileconditionsofthesametypearecombinedwithOR.
www.it-ebooks.info
ChoosingactionoperationsDiscoveryactionsaresomewhatsimplerthantriggeractionsastherearenostepsorescalationsinvolved.Thisdoesn’tmeanthatyoudon’thavequiteafewoptionstochoosefrom:
Pleasenotethatevenifyoudefinedadefaultmessage,itwon’tbesentuntilyouspecifytherecipientsinthissectionusingtheSendmessageoperation.Ontheotherhand,ifadding(orremoving)ahostisaquiteself-explanatoryaction,whenitcomestoaddingtoahostgrouporlinkingtoatemplate,itbecomesclearthatagoodsetofactionswithspecificreceivedvalueconditionsandtemplate-linkingoperationscangiveahighlevelofautomationtoyourZabbixinstallation.
NoteThishighlevelofautomationisprobablymoreusefulinrapidlychangingenvironmentsthatstilldisplayagoodlevelofpredictability,forexample,thekindofhostsyoucanfind,suchasfast-growinggridsorclusters.Inthesekindsofenvironments,youcanhavenewhostsappearingonadailybasis,andmaybeoldhostsdisappearatalmostthesamerate,butthekindofhostismoreorlessalwaysthesame.Thisistheidealpremiseforasmallsetofwell-configureddiscoveryrulesandactions,soyoudon’thavetoconstantlyandmanuallyaddorremovethesametypesofhosts.Ontheotherhand,ifyourenvironmentisquitestableoryouhaveaveryhighhosttypevariability,youmightwanttolookmorecloselyatwhich,andhowmanyhosts,youaremonitoringasanyerrorcanbemuchmorecriticalinsuchenvironments.
Also,limitingdiscoveryactionstosendingmessagesaboutdiscoveredhostscanprove
www.it-ebooks.info
quiteusefulinsuchchaoticenvironmentsorwhereyoudon’tcontroldirectlyyoursystems’inventoryanddeployment.Insuchcases,gettingsimplealertsaboutnewhosts,ordisappearingones,canhelpthemonitoringteamkeepZabbixupdateddespiteanycommunicationfailurebetweenITdepartments,accidentalorotherwise.
Moreover,youarenotstuckwithe-mailsandSMSesfornotificationsorlogging.InanActionoperationform,youcanonlychooserecipientsasZabbixusersandgroups.Iftheusersdon’thaveanymediadefined,ortheydon’thavetherightmediafortheactionoperation,theywon’treceiveanymessage.AddingmediatousersisdonethroughtheAdministrationtaboftheZabbixfrontend,whereyoucanalsospecifyatimewindowforaspecificmediatobeused(sothatyouwon’tgetdiscoverymessagesasanSMSinthemiddleofthenightforexample).Speakingofusersandmediatypes,youcanalsodefinecustomones,throughtheMediatypessectionoftheAdministrationtabinZabbix’sfrontend.NewmediatypeswillbeavailablebothintheMediasectionoftheuserconfigurationandastargetsformessagesendingintheActionoperationsform.
AninterestingusefornewmediatypesistodefinecustomscriptsthatcangobeyondsimpleemailorSMSsending.
AcustommediascripthastoresideontheZabbixserver,inthedirectoryindicatedbytheAlertScriptsPathvariable,inthezabbix_server.confconfigurationfile.Whencalledupon,itwillbeexecutedwiththreeparameterspassedbytheserverandtakenfromtheactionconfigurationinthecontextoftheeventthatwasgenerated:
$1:Thisistherecipientofthemessage$2:Thisisthesubjectofthemessage$3:Thisisthemainmessagebody
Therecipient’saddresswillbetheonedefinedforthenewmediatypeinthecorrespondingmediapropertyfortheuserspecifiedintheactionoperationstep.Thesubjectandthemessagebodywillalsobepassedaccordingtotheactionoperationstep,asshownintheprecedinglist.ThisisallthatZabbixneedstoknowaboutthescript.
Thefactis,acustomscriptcanactuallydomanydifferentthingswiththemessage:loggingtoalocalorremotedirectory,creatinganXMLdocumentandinteractingwithalogmanagerwebservicesAPI,printingonacustomdisplay—justaswitheverycustomsolution,thesky’sthelimitwithcustommediatypes.
Hereisasimple,practicalexampleofsuchacustommediatype.Let’ssaythatyourITdepartmenthasimplementedaself-provisioningserviceforvirtualmachinessothatdevelopersandsystemadminscancreatetheirownVMsandusethemforalimitedamountoftimebeforetheyaredestroyedandtheresourcesrecycled.Thislaboratoryofsortshasbeenputinaseparatenetwork,butusersstillhavetogainaccesstoit,andtheyarealsoadministratorsofthoseVMs,sothere’sverylittlecontroloverwhatgetsinstalled,configured,oruninstalledonthosemachines.Inotherwords,whileyoucouldprovisiontheVMswithapreinstalledZabbixagent,youcan’treallyrelyonthefactthatyourusers,whetherinadvertentlyorforspecificreasons,wouldnotdisableit,orwouldnotinstallservicesthatshouldreallynotbethere,likeaDHCPserverforexample.So,youdecideto
www.it-ebooks.info
keepaneyeonthosemachinesdirectlyfromtheZabbixserver(orasuitableproxy)andimplementasimplediscoveryrulethatwillgenerateadiscoveryeventforeveryhostthatrespondstoanICMPechorequestandnothingmore,asfollows:
Basedonthatrule,you’llwanttoconfigureanactionthat,foreveryhostinthatsubnet,willperformaportscanandreporttheresultsviamailtoyou.
Todothat,you’llfirstneedtohaveacustommediatypeandthecorrespondingscript.So,youheadtoAdministration|MediatypesandclickonCreatemediatype.Oncethere,youassignasuitablename,selectScriptasatypeandprovideZabbixwiththenameofthescripttoexecute.Here,youjustneedtodefinethescriptname,asshowninthefollowingscreenshot.You’llfindoutlaterinthechapterinwhatdirectorytheactualscriptshouldbeplaced:
Justaddingamediatypeisnotenoughthough,you’llhavetoenableitfortheuseryouintendtosendthosereportsto.JustheadtoAdministration|Usersandselecttheuseryouwanttoaddthenewmediatypeto.Quitepredictably,thetabyouwantiscalledMedia.Addthemediayoujustcreatedandremembertoalsoaddawaytotellthescript
www.it-ebooks.info
whereitshouldsendtheresults.Sinceyouareinterestedinreceivingane-mailaddressafterall,that’swhatwe’lltellZabbix,asfollows:
TheSendtoparameterwillbethefirstargumentpassedtoport_scan.sh,followedbythesubjectandthebodyofthemessagetosend.So,beforeactuallydeployingthescript,let’sdefinethesubjectandthebodyofthemessage.Todothat,you’llneedtocreateanactionforthediscoveryevent,asfollows:
Forthepurposesofthescript,allyoureallyneedistheIPaddressofthehostyouaregoingtoscan,butitcertainlywouldn’thurttoaddsomemoreinformationinthefinalmessage.
www.it-ebooks.info
Thenextstepistodefinesomeconditionsfortheaction.Rememberthatactionsareglobal,sothefirstconditionyouwanttosetistheIPrangeonwhichthisactionwillbeperformed,otherwiseyou’druntheriskofperformingaportscanoneverydiscoveredhostinyournetwork.
Youmightalsowanttolimittheactionasaconsequenceforthediscoveryruleyoucreated,independentofanyotherrulesyoumighthaveonthesamenetwork.
Finally,youshouldmakeadecisionaboutthediscoverystatus.Ifyouwantaperiodicupdateofwhatportsareopenonadiscoveredhost,you’llalsoneedtodefineaconditionforthehosttobeUp:inotherwords,forthehosttobereportedasliveforatleasttwoconsecutivechecks.
Foraslongasthehoststaysup,aportscanwillbeexecutedandreportedaccordingtothediscoveryintervaloftheruleyoudefinedearlier.Ifyoujustwantaportscanforanewhostorforahostthathasbeenreportedasdownforawhile,you’lljustneedtofiretheactionontheconditionthatthehostisDiscovered;thatis,itisnowbeingreportedup,whileitwasdownbefore.Whatiscertainisthatyou’llwanttoavoidanyactionifthehostisdownorunavailable.
Thefollowingscreenshotencapsulatesthediscussioninthisparagraph:
Thelaststepistodefinetheactionoperationthatissendingthemessageviatheport_scancustommediatypetotheuseryouwant,asfollows:
www.it-ebooks.info
Oncedonewiththis,youarefinallyreadytocreatetheport_scan.shscript.So,headtotheAlertScriptsPathdirectoryasconfiguredinyourzabbix_server.conf(it’susuallydefinedas/usr/lib/zabbix/alertscripts)andcreatethefollowingscriptthere:
#!/bin/bash
RECIPIENT=$1
IPADDRESS=$2
MESSAGE=$3
SCAN="nmap-AT5-sT"
RESULT=$($SCAN$IPADDRESS)
(echo"ScanresultsforIP$IPADDRESS";
echo"$RESULT";
echo"";
echo"$MESSAGE")|mailx-s"Scanresultsfor$IPADDRESS"$RECIPIENT
NoteDon’tforgettosetthecorrectownershipandpermissionsforthescriptonceyouaredone:
#chownzabbixport_scan.sh
#chmod755port_scan.sh
Asyoucansee,theprogramthatwillperformtheactualportscanisNmap,somakesureyouhaveitinstalled.Incaseyoudon’thaveitinstalled,asimpleyuminstallnmapwilltakecareofthat.TheoptionspassedtoNmaparejustthebasics:-sTperformsasimpleconnect()scan.It’snotthefanciestone,butit’stheonlyoneavailabletonon-rootusers,
www.it-ebooks.info
andthescriptwillbeexecutedbyZabbixasthezabbixuser.–Aturnsontraceroute,OS,andservicedetectionsothattheoutputisascompleteaspossible.Finally,-T5forcesNmaptoexecutetheportscaninaslittletimeaspossible.Oncethescripthastheresultsoftheportscan,itwilljustconstructthemessageandsendittotherecipientdefinedintheaction.
Thisis,ofcourse,averybasicscript,butitwillgetthejobdone,andyou’llsoonreceiveaportscanreportforeverynewVMcreatedinyourself-provisioninglab.Tokeepthingssimpleandclear,wedidnotincludeanyconsistencycheckingorerrorreportingincaseofproblems,sothat’scertainlyawayyoucanimproveonthisexample.Youcouldalsotrytosendtheresultstoalogfile(oralogdirectory)insteadofamailaddress,oreventoadatabase,sothatotherautomationcomponentscanpickupthereportsandmakethemavailableviaothermediasuchaswebpages.Whatyou’llprobablywanttoavoidistodirectlychangethehost’sconfiguration,orZabbix’sownone,throughthisscript.
Evenifnoonewillpreventyoufromdoingso,it’sprobablybestifyouavoidusingallthispowertoexecutecomplexscriptsthatmightchangeyournetworkconfiguration,suchasenablinginterfaces,addingrulestoafirewall,andsuchlike.Whilethisisperfectlypossibleusingacustommediascript,thisshouldbethedomainofremotecommands.Thesewilltakecenterstageinthenextparagraph.
RemotecommandsTherearequiteafewoptionsavailabletoyouwhenitcomestoexecutingremotecommandsasanactionoperation.
YoucandefinealistofIPMIcommandstoberunonthetargethostoraseriesofSSHcommandsthatconnecttoaboxandperformvariousoperationsthere.AremotecommandcouldevenbeasimplewrapperforaremotescriptdeployedonaZabbixagent,oracustomscriptthatwillberuneitheronanagentorontheZabbixserveritself.
Thetruthis,sometimes,remotecommandscanbejustalittletoopowerful.Youcanstartandstopservices,deployorprovisionsoftware,makeconfigurationchanges,openorclosefirewallports,andeverythingelseyoucanpossiblyimagine,aslongasyoucanwriteascriptforit.Whilethiscansoundfascinatingandpromising,wehavefoundovertheyearsthatthesesolutionstendtobefragileandunpredictable.OneofthereasonsisthatZabbixdoesn’twarnyouifaremotecommandfails.Moreimportantly,environmentstendtochangefasterthantheseautomationtoolssothatyoucanquicklyfindyourselfdealingwiththeunintendedconsequencesofaremotecommandrunningwhereitshouldnotrun,ornotrunningwhenitshouldrun.
Themoreoftheseyouadd,themoreitwillbehardtokeeptrackofthem,andthemoreonecanbeluredintoafalsesenseofsecurity,countingonthefactthatremotecommandsaretakingcareofthings,while,infact,theymaybecontributingtothechaosinsteadoftamingit.
Thatsaid,it’scertainlyundeniablethatremotecommandscanbeuseful.Let’sseeanexamplethatisbothhelpfulforyourZabbixconfigurationandalsofairlysafe.
www.it-ebooks.info
InChapter2,ActiveMonitoringofYourDevices,we’veseenhowit’spossibletousesomeofthemeasurements,asreportedbyahost’sitems,topopulatethesamehost’sinventoryfields.Thisisagreatsolutionforthefieldsthatcanbefilledthisway,butwhatabouttheotherones?ThingslikePOCdetails,maintenancedates,installername,installedsoftware,andsuchlikecan’talwaysbeextrapolatedfrommonitoringmetricsastheymaysimplynotbeavailableonthemonitoredhostitself.
Theyusuallyareavailable,though,onassetinventorysystemsthatITdepartmentsusetokeeptrackofavailableresources.
Inthefollowingexample,you’llcreateanactionoperationthatwillexecutearemotecommandontheZabbixserver,fetchsomeinventoryinformationfromanassetdatabase,andfilluporupdatethehost’sinventorydetails.
Beforeproceedingwiththecommand,let’smakeanassumptionandsomepreparations.
Therearemanyassetinventorysystemsavailable,someproprietaryandsomeopensource.Allofthemhavedifferentdatabaseschemasanddifferentwaystoexposetheirdata.Moreover,aninventorydatabasestructuredependsasmuchontheactualenvironmentit’sputinto,andtheprocessesthatgoverntheaforesaidenvironment,asitisonitsinternalspecifications.So,wedecidedtouseadummyassetmanagementtoolthatwillreturn,givenanIPaddress,asimpleJSONobjectcontainingalltheinventorydatayouneedforthetaskathand.Theassumptionisthatyou’llbeabletoputtheexampleintoyourcontextandfigureouthowtoextractthesameinformationfromyourowninventorymanagementsystem,andthatyouwillalsoknowwhatauthenticationschemeyouwillrelyonifyouneedtomakejustonerequestormultiplerelatedrequests,andsoon.
Secondly,forpracticalreasonswearegoingtousePythonasthelanguageofthecommandscript,soyou’llwanttomakesurethatit’sinstalledandavailableonyourZabbixserver.Ifit’snotthere,youcaninstallit,andtherelatedutilities,quiteeasilyusingyum:
#yuminstallpython
#yuminstallpython-setuptools
#easy_installpip
Finally,wearegoingtointeractwithZabbix’sconfigurationnotthroughdirectqueriestoitsdatabase,butthroughitsAPI.Inordertodothat,we’lluseaveryusefulPythonlibrary,calledpyzabbix.Youcanfinditathttps://github.com/lukecyca/pyzabbix,butsinceyouinstalledpip,itwillbeextremelyeasytomakeitavailabletoyourPythoninstallation.Justrunthefollowingcommand:
#pipinstallpyzabbix
ThePythonpackagemanagerwilldownloadandinstallitforyou.
Nowwearereadytoconfigurethediscoveryactionandwritetheactualcommandscript.
Youcanchoosetoreuseanexistingdiscoveryrule,suchasthesimpleICMPruleyouusedinthepreviousparagraph,youcancreateanewonespecifictoasinglenetworktoscan,asingleTCPportthathastobeavailable,orthepresenceofaZabbixagent.Wewon’tgo
www.it-ebooks.info
intoanymoredetailshere,asyou’vealreadylearnedhowtoconfigureoneearlierinthechapter.Similarly,wecansafelyskipanydetailabouttheactionconditionsastheymightalsobeentirelysimilartothoseshownearlier.Whatchangesis,ofcourse,theactionoperation.Thefollowingscreenshotwillgiveyouabetterideaofwhatwehavebeentalkingaboutinthisparagraph:
TheimportantelementsherearethefactthatthescriptshouldbeexecutedontheZabbixserver,thefactthatwespecifiedthefullpathforthescript,andthefactthatweareusingthe{DISCOVERY.IPADDRESS}macroastheargument.
Oncetheactionisconfigured,youarereadytopreparetheactualscript.Let’sseehowitwouldlook:
#!/usr/bin/python
importsys
importjson
frompyzabbiximportZabbixAPI
importdummy_inventory_api
ipaddr=sys.argv[1]
hostinfo_json=dummy_inventory_api.getinfo(ipaddr)
#hostinfo_jsonwillcontainaJSONstringsimilartothisone:
#{"hostip":"172.16.11.11",
#"hostname":"HostA",
www.it-ebooks.info
#"inventory":{
#"asset_tag":"12345678",
#"install_date":"31-11-2014",
#"installer_name":"SKL"
#}
#}
hostinv=json.loads(hostinfo_json)['inventory']
zbx=ZabbixAPI(http://127.0.0.1/zabbix/)
zbx.login("admin","zabbix")
hostinfo=zbx.host.get(output=['hostid'],filter={'ip':ipaddr})
hid=hostinfo[0]['hostid]
zbx_inventory={
'date_hw_install':hostinv['install_date'],
'installer_name':hostinv['installer_name'],
'asset_tag':'12345678'
#addotherfieldsyoumaybeinterestedin…
}
zbx.host.update(hostid=hid,inventory=zbx_inventory)
sys.exit()
Asyoucansee,thescriptisfairlystraightforwardandsimplistic,butitcanbeusedasastartingpointforyourowninventory-updatingscripts.Themainthingthatyouneedtotakecareofistofigureouthowtogetyourinventorydatafromyourassetdatabase.YoumightneedtoconnecttoaRESTAPI,orgetanXMLdocumentviaawebservice,orevenperformsomequeriesviaODBC.WhatmattersisthatyouendupwithaPythondictionaryorlistcontainingallthatyouneedtoupdatetherelevanthostinZabbix.
ThesecondpartofthescriptfirstofallshowsyouhowtoconnecttotheZabbixAPIusingtheZabbixAPIconstructor.Itthenproceedswiththeloginmethod,whereyou’llneedtoprovidethecredentialsyouconfiguredearlier.
Allgetmethodsacceptafilterparameterthatyoucanusetoretrieveasingleobjectoralistofobjectsthatsatisfycertainconditions.Inthiscase,weusedittogetthehostidofthehostthatisassociatedwithaspecificIPaddress.
Payattentiontothenextlineasthevaluereturnedbyallgetmethodsisalwaysalist,evenifitcontainsonlyoneelement.That’swhyweneedtoreferencethefirstelementofhostinfo,element0,beforereferencingtheinventorydictionarykey.
Weonlyshowedthreeinventoryfieldshere,buttherearemanymoreavailableinZabbix,soitmaybeagoodideatobuildadictionarywithallZabbixinventoryfieldsaskeysandtheretrievedvaluesasvalues.
Nowthatwehavethehostidandtheinventoryinformationatourdisposal,wecanproceedwiththeactualinventoryupdate.Theupdatemethodisfairlystraightforward:youspecifythehostidofthehostyouwanttoupdateandthenewvaluesforthefieldsthatyouneedtoupdate.
Andthat’sit,withascriptlikethisconfiguredasaremotecommandforadiscoveryaction,youcankeepyourZabbixinventorydatainsyncwithwhateverassetmanagementsystemyoumayhave.
www.it-ebooks.info
Asyoumighthaverealized,hostdiscoverycanbequiteacomplexmatterbecauseofthesheernumberofvariablesyouneedtotakecareof,andbecauseit’snotalwayseasy,inareal-worldnetwork,toidentifyaclearlogicforhostcreation,templateassignment,andothermonitoringparameters,basedondiscoverydata.
Low-leveldiscovery,bycontrast,ismuchmoresimple,givenitspowertodynamicallycreatespecificitemsasahost’savailableresourcesarediscovered.So,let’susetheremainingpagesofthischaptertoexploreafewaspectsofthisextremelyusefulfeature.
www.it-ebooks.info
Low-leveldiscoveryAnextremelyusefulandimportantfeatureofZabbixtemplatesistheirabilitytosupportspecialkindsofitemscalledlow-leveldiscoveryrules.Onceappliedtoactualhosts,theseruleswillquerythehostforwhateverkindofresourcestheyareconfiguredtolookfor:filesystems,networkinterfaces,SNMPOIDs,andmore.Foreveryresourcefound,theserverwilldynamicallycreateitems,triggers,andgraphsaccordingtospecialentityprototypesconnectedtothediscoveryrules.
Thegreatadvantageoflow-leveldiscoveryrulesisthattheytakecareofthemorevariablepartsofamonitoredhost,suchasthetypeandnumberofnetworkinterfaces,inadynamicandgeneralway.Thismeansthat,insteadofmanuallycreatingspecificitemsandtriggersofeveryhost’snetworkinterfacesorfilesystems,orcreatinghugetemplateswithanypossiblekindofitemforaparticularoperatingsystemandkeepingmostoftheseitemsdisabled,youcanhaveareasonablenumberofgeneraltemplatesthatwilladaptthemselvestothespecificsofanygivenhostbycreatingontheflyanyentityrequired,basedondiscoveredresourcesandpreviouslyconfiguredprototypes.
Outofthebox,Zabbixsupportsfourdiscoveryrules:
NetworkinterfacesFilesystems’typesSNMPOIDsCPUsandCPUcores(asofversion2.4)
Asdiscoveryrulesareeffectivelyspecialkindsofitems,youcancreateyourownrules,providedyouunderstandtheirpeculiaritycomparedtoregularitems.
Youneedtocreateandmanagelow-leveldiscoveryrulesintheDiscoveryrulessectionofatemplateconfigurationandnotintheusualItemssection,evenifthediscoveryrulesendupcreatingsomekindofitems.Themaindifferencebetweendiscoveredandregularitemsisthat,whereasaregularitemusuallyreturnsasinglevalue,adiscoveryitemalwaysreturnsalist,expressedinJSON,ofmacrovaluepairs.Thislistrepresentsalltheresourcesfoundbythediscoveryitems,togetherwithameanstoreferencethem.
ThefollowingtableshowsZabbix’ssupporteddiscoveryitemsandtheirreturnvalues,togetherwithageneralizationthatshouldgiveyouanideaofhowtocreateyourownrules:
Discoveryitemkey Itemtype Returnvalues
vfs.fs.discovery Zabbixagent
{"data":[
{"{#FSNAME}":<path>","{#FSTYPE}":"<fstype>"},
{"{#FSNAME}":<path>","{#FSTYPE}":"<fstype>"},
{"{#FSNAME}":<path>","{#FSTYPE}":"<fstype>"},
…
]}
{"data":[
{"{#IFNAME}":"<name>"},
{"{#IFNAME}":"<name>"},
www.it-ebooks.info
net.if.discovery Zabbixagent {"{#IFNAME}":"<name>"},
…
]}
snmp.discovery SNMP(v1,v2,orv3)agent
{"data":[
{"{#SNMPINDEX}":"<idx>","{#SNMPVALUE}":"<value>},
{"{#SNMPINDEX}":"<idx>","{#SNMPVALUE}":"<value>},
{"{#SNMPINDEX}":"<idx>","{#SNMPVALUE}":"<value>},
…
]}
system.cpu.discovery Zabbixagent
{"data":[
{""{#CPU.NUMBER}":"<idx>","{#CPU.STATUS}":"<value>},
{"{#CPU.NUMBER}":"<idx>","{#CPU.STATUS}":"<value>},
{"{#CPU.NUMBER}":"<idx>","{#CPU.STATUS}":"<value>},
…
]}
custom.discovery Any
{"data":[
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":"<value>"},
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":"<value>"},
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":"<value>"},
…
]}
TipJustaswithallSNMPitems,theitemkeyisnotreallyimportantaslongasitisunique.It’stheSNMPOIDvaluethatyouaskanagentforthatmakesthedifference:youcancreatedifferentSNMPdiscoveryrulesthatlookfordifferentkindsofresourcesbychangingtheitemkeyandlookingfordifferentOIDvalues.Thecustomdiscoveryexampleisevenmoreabstractasitwilldependontheactualitemtype.
Asyoucansee,adiscoveryitemalwaysreturnsalistofvalues,buttheactualcontentsofthelistchange,dependingonwhatresourcesyouarelookingfor.Inthecaseofafilesystem,thereturnedlistwillcontainvalueslike{#FSNAME}:"/usr",{#FSTYPE}:"btrfs",andsoonforeverydiscoveredfilesystem.Ontheotherhand,anetworkdiscoveryrulewillreturnalistofthenamesofthediscoverednetworkinterfaces.ThisisthecaseforthedefaultSNMPnetworkinterfacestemplate.Let’sseeindetailhowitworks.
Thetemplatehasadiscoveryrulecallednetworkinterfaces.Itlooksjustlikearegularitemasithasaname,atype,anupdateinterval,andakey.It’sanSNMPtype,soitalsohasanSNMPOID,IF-MIB::ifDescr.Thisisadiscoveryrule,soinsteadofasinglevalue,itwillreturnalistofalltheOIDsthatarepartoftheIF-MIB::ifDescrsubtreeforthatparticulardevice.ThismeansthatitwillreturntheOIDanditsvalueforallthenetworkinterfacespresentonthedevice.Everytimethediscoveryruleisexecutedonahost(basedontheupdateinterval,justlikeanyotheritem),itwillreturnalistofallinterfacesthatareavailableatthatparticularmoment.Ifthedevicehadfournetworkinterfaces,itcouldreturnsomethingsimilartothis:
{"data":[
{"{#SNMPINDEX}":"1",
"{#SNMPVALUE}":"FastEthernet0/0"},
{"{#SNMPINDEX}":"2",
www.it-ebooks.info
"{#SNMPVALUE}":"FastEthernet0/1"},
{"{#SNMPINDEX}":"3",
"{#SNMPVALUE}":"FastEthernet1/0"},
{"{#SNMPINDEX}":"4",
"{#SNMPVALUE}":"FastEthernet1/1"},
]}
Thediscoveryrulewillthenproceedtoapplythelisttotheitemandtriggerprototypesithasconfigured,asfollows:
TakingtheIncomingtrafficoninterface{#SNMPVALUE}itemprototypeasanexample,youcanseehowitallcomestogether:
The{#SNMPVALUE}macroisusedintheitem’skeyand,therefore,intheitem’snameaswell(lookatthe$1macrothatreferencesthefirstargumentoftheitem’skey).
www.it-ebooks.info
Ontheotherhand,the{#SNMPINDEX}macrowillbeusedbyZabbixtoactuallygettheincomingtrafficvalueforthatspecificinterfaceasitshouldbeclearbynowifyouobservethevalueintheSNMPOIDfield.
Whenconfiguringatemplate’sdiscoveryrules,youdon’tneedtocareabouttheactualvaluesreturnedintheirlists,northelists’length.Theonlythingyouhavetoknowisthenameofthemacrosthatyoucanreferenceinyourprototypes.Thesearetobereferencedinthesecondhalfofthelow-leveldiscoverymechanism,objectprototypes.Youcreatethemasregulartemplateentities,makingsureyouusethediscoveryitemmacroswhereneeded,andZabbixwilltakecareoftherestforyou,creatingforeachitemprototypeasmanyitemsasthereareelementsinthelistreturnedbythediscoveryrule,foreachtriggerprototypeasmanytriggersasthereareelementsinthelistreturned,andsoon.
So,whenyouapplythetemplatetoahost,itwillcreateitems,triggers,andgraphsbasedontheresourcesdiscoveredbythediscoveryitemsandconfiguredaccordingtothediscoveryprototypes.
Customdiscoveryrules,fromthispointofview,workexactlyinthesamewayascustomitems,whetheryoudecidetouseagent-sidescripts(therebyusingacustomzabbix.agentitemkey),externalscripts,databasequeries,oranythingelse.Theonlythingsyouhavetomakesureofisthatyourcustomitemsreturnkeys/valuesthatfollowtheJSONsyntax,asshownintheprecedingtable,andthatyoureferenceyourcustommacrosintheentitiesprototypesthatyouwillcreate.
Let’sseeanexampleofacustomdiscoveryruleusingagainNmapanditsoutputtodynamicallycreatesomeitemsforahost,representingtheopenportithas,andthekindofservicesthatarelistening.WhywouldyouwanttouseNmapandaportscan?Thedeviceyouneedtomonitormaybedoesn’tsupporttheZabbixagent,soifyoujustaskfortheoutputofnetstat,youmightnotbeabletoinstalltheagentforadministrativereasons,oryoumighthavetomakesurethattheservicesarealsoavailablefromanothernetwork,socheckingthemfromafar,insteadofdirectlyonthehost,willenableyoutoalsoverifyyourfirewallrules,killingtwobirdswithonestone.
Eitherway,we’llcreateanexternalcheckitemperopenTCPport,configuredasacharacter-typeitem.Eachitemwillcontainthenameoftheservicethatwasfoundlistening,ifany,asreportedbyNmap’sservicediscoveryfacilities.
Startbycreatingthediscoveryruleasanexternalcheckthatwillcallaport-mappingscript,asfollows:
www.it-ebooks.info
Asyoucansee,thescriptwillreceivethehost’sIPastheonlyargument,anditwillrunonceanhourforeveryhostthathasthisdiscoveryruleconfiguredandisactive.
ThescriptitselfisverysimpleandisbasedonNMAP’sXMLoutputcoupledwiththeniftyxml2toolyoualreadyusedinChapter3,MonitoringYourNetworkServices,asfollows:
#!/bin/bash
IPADDR=$1
#storeportsasarray
PORTS=($(nmap-sV-oX-${IPADDR}|xml2|grepportid|cut-d'='-f2))
#countelementsofthearrayanduseascounterforlaterprocessing
COUNTER=${#PORTS[@]}
#openJSON
echo'{"data":['
#loopthroughportsandprintkey/value
forPORTin"${PORTS[@]}";do
COUNTER=$((COUNTER-1))
if[$COUNTER-ne0];then
echo"{\"{#PORTID}\":\"${PORT}\"}",
else
#it'sthelastelement.TohavevalidJSONWedon'taddatrailingcomma
echo"{\"{#PORTID}\":\"${PORT}\"}"
fi
done
#closeJSON
echo]}
#exitwithcleanexitcode
exit0
Thelinestartingwithnmapistheheartofthescript.The–oXoptionenablesXMLoutput,whichismorestableandeasytomanagecomparedtothenormalone.Thedashafter–oXspecifiesstdoutastheoutputinsteadofaregularfile,sowecanpipetheresulttoxml2andthentakeonlythelinesthatcontainportid,thatis,theopenportnumbersforthathost.
www.it-ebooks.info
Asaresult,thescriptjustoutputsasimpleJSONobject.Here’sanexampleofwhatthediscoveryrulewillget,asshownfromthecommandline:
./port_map.sh'127.0.0.1'
{"data":[
{"{#PORTID}":"22"},
{"{#PORTID}":"25"},
{"{#PORTID}":"80"},
{"{#PORTID}":"631"},
{"{#PORTID}":"3306"}
]}
It’snowtimetodefinetheitemandtriggerprototypes,basedontheopenportthatyoufound.We’llshowhereanexampleofanitemprototypethatwillreturnthenameandversionofthedaemonlisteningontheport,asreturned,onceagain,byNmap:
Theexternalcheckwillcallascriptthatisevensimplerthanthepreviousone,asfollows:
#!/bin/bash
IPADDR=$1
PORT=$2
nmap-sV-oX--p${PORT}${IPADDR}|xml2|grep'port/service/@\
(product\|version\|extrainfo\)'
ComparedtothepreviousNmapcommand,weaddeda–sVoptiontomakeNMAPrunaseriesofprobesinordertofindoutwhatserviceisrunningbehindthatopenportanda–poptiontospecifyasingleporttoscan.
Theoutputwaskeptsimpleonpurposetoshowyouanexampleofxml2’soutput.Youcan,ofcourse,sliceitanddiceittosuityourownneeds:
./port_service.sh127.0.0.180
/nmaprun/host/ports/port/service/@product=Apachehttpd
/nmaprun/host/ports/port/service/@version=2.2.15
/nmaprun/host/ports/port/service/@extrainfo=(CentOS)
NoteTheamountofinformationNmapwillbeabletogetfromanetworkservicedependsvery
www.it-ebooks.info
muchonhowmuchandonwhatkindofdatatheserviceisconfiguredtoexpose.Thismightdependonbuilt-inparametersorsecurityconsiderationsonthepartoftheserviceowner.Comparedtothepreviousexample,yourmileagecanvary.
Thisiswhatwillappearasthevalueoftheitemoncethediscoveryruleisactivated.
www.it-ebooks.info
SummaryInthischapter,youlearnedhowtouseZabbix’sdiscoveryfacilitiestoautomateitsconfigurationasmuchaspossible.Itshouldalsobecleartoyouwhyit’simportanttominimizethedifferencebetweenwhatisconfiguredinZabbixandwhatisactuallyoutthereonthewire.Keepingtrackofeverythingthatcanappearordisappearonabusynetworkcanbeafulltimejobandonethatisbettersuitedtoautomatedmonitoringfacilitieslikethisone.Younowhavealltheskillsneededtoactuallydoit,andyouarereadytoapplytheminyourreal-worldenvironment.
Inthenextchapter,we’llwrapthingsupbyshowingyouhowtoleverageZabbix’spresentationpowertocreateandmanagegraphs,dynamicmaps,andscreens.
www.it-ebooks.info
Chapter5.VisualizingYourTopologywithMapsandGraphsAsyouprobablyalreadyknow,Zabbix’sapproachtomonitoringisbasedonseparatingdatagatheredfromtriggerlogicandeventlogging.Ontheonehand,thismeansthatyouareabletoreferenceanymeasurement,presentandpast,inyourtriggers,makingthemallthemorepowerful.Ontheotherhand,italsomeansthatyouhavedirectaccesstoallyourmeasurementhistoryforallyouritems.
Whilesortingthroughallofyourhistoricaldatatolookforaspecificvaluecancertainlybeuseful,therealadvantagehereistoleverageZabbix’sgraphingandmappingfunctionalitiestoaggregateandvisualizedatainmeaningfulways.
Inthischapter,you’llseehowtocreatecomplexgraphsfromyouritems’numericalvalues,howtoautomaticallydrawmapsthatreflectthecurrentstatusofyournetwork,andhowtobringitalltogetherusingscreensasatooltocustomizemonitoringdatapresentation.
www.it-ebooks.info
CreatingcustomgraphsBasicgraphicaldatarepresentationcomesforfreeforanyitemthathasanumericdatatype.YoujustneedtogotoMonitoring|LatestData,selectthehostyouareinterestedin,findtherelevantitem,andclickonGraphinthelastcolumnontheright-handside.You’llgetalinegraphwithatimesliderthatyoucanusetochangethetimeframeofthegraphitself;widenittocoveralongeramountoftime,orshortenittofocusonaspecificpointintime.
SinceZabbix2.4,youcanalsocomparedifferentitemsontheflywithadhocgraphs.Theseareadirectextensionofsimplegraphs:fromMonitoring|LatestData,youjustneedtomarkthecheckboxontheleft-handsideofeveryitemthatyouwanttographandselectDisplaystackedgraphorDisplaygraphfromthedrop-downmenuatthebottomofthepage,asfollows:
Theresultisprettymuchtheoneyouexpect.Youalsodon’thavetoworrytoomuchaboutchoosingbetweenanormalgraphandastackedgraphasyou’llbeabletoswitchbetweenthetwofromthegraphitself,asfollows:
Thesequick,adhocgraphscanreallycovermostofyourvisualizationneeds,especiallyforvaluesthatyoudon’tconsultthatoftenorifyouneedtocompareitemsthatyou
www.it-ebooks.info
normallydon’thaveto,aspartofanewanalysisortoinvestigateanewclassofproblems.
Ontheotherhand,ifyouneedtocomparethesametypesofitemsoverandover,andfordifferenthosts,you’llneedawaytosaveyourselectionssothatyouareabletoaccessyouraggregatedgraphswithouthavingtospecifyeverytimewhatitemsneedtobegraphed.Youcanachieveallthiswithcustomgraphs.
NoteIfyouliketovisualizeyourpercentiledatawithpiecharts,you’llalsoneedtocreatecustomgraphsasthey’recurrentlytheonlywaytocreatepiechartsinZabbix.
Customgraphscanbecreatedaspartofahost,orbetteryetaspartofatemplate,oralow-leveldiscoveryrule,sothatanyhostinheritingthetemplateordiscoveryrulewillautomaticallyalsoinheritthecustomgraph.
Tocreateone,youneedtogotoConfiguration|Templates,choosethetemplateyouwanttoputyourgraphinto,selectGraphs,andclickonCreategraph.Thiswillbringyoutothegraphcreationform.Forconvenience,thefollowingexamplewillshowyousomeitemsalreadyaddedtotheitemlistandsomeotheroptionsalreadyselectedinsteadofanemptyform,butyou’lleasilybeabletoaddyourownitemsbyfollowingtheaddlinkatthebottomoftheitemlist,asfollows:
Asyoucansee,thereareafewoptionsworthnoting.Firstofall,youcanselectthegraphtypebetweenNormal,Stacked,Pie,andExploded(thatis,apiechartwithallslicesseparatedinsteadofclosetogether).Next,ifyouselecttheShowtriggerscheckbox,thegraphwillincludeahorizontallineforeverytriggerthathasanyoftheitemspresentin
www.it-ebooks.info
thegraph’sitemlistinitsexpression.Youdon’thavetospecifythetriggerorfindthemmanually;Zabbixwilltakecareoffindingallrelevanttriggersandshowthemonthegraph.
Youcanalsospecifytherangeofyaxisvalueseitherasfixedvaluesorcalculatedbasedonthedatayouhave.You’llnormallywanttosetthemascalculatedasthisoptionwillusuallyshowtheclearestandbest-lookinggraphs,butsometimes,youmightwanttosetthemtoafixedvaluetohaveabetterunderstandingofhowthevalueschange,especiallyiftheyfluctuatealotbetweenverybigandverysmallvalues,andtheitemexpressesapercentilerange.
Movingtotheitemlist,youcanordertheitemsbydragginganddroppingthebluearrowsontheleft-handsideoftheitem’snameandchangetheircolorbyeitherspecifyinganRGBvalueorchoosingfromacolorpalette.
Thedrawstylecanbequiteusefulifyouwantaspecificitemtostandoutfromtherest.Therearequiteafewstylesavailableforanormalgraph,whilethisoptionisnotavailableforstackedandpiecharts.
TheFunctiondrop-downmenuenablesyoutochoosehowtheitemshouldbegraphedforeverytickinthexaxis:youcanchoosebetweentheminimumvalue,themaximumone,andtheaverage.Keepinmindthatthex-axistickdensitywillchangedynamicallywiththetimescaleofthegraph(youcanselectdifferenttimeframeswhilelookingatagraph;youdon’thavetospecifytheminadvance):fortimeframesuptoanhour,itwillshoweverysamplecollected,dependingontheitems’samplefrequency;forlargertimeframes,you’llhavex-axisticksproportionaltothetimeframeselected,whichisafewminutesiftheglobaltimeframeisafewhours,todaysorweeksifyouselectmonths’oryears’worthofmonitoringdata.Foreverytick,Zabbixwillusethefunctionyouselectedheretoplottheitemvalueeitherbyselectingthemaximum,theminimum,ortheaveragevalueforthattimetick.
Finally,youcanchoosewhethertheyaxisforanitemwillbeshownontheleft-handsideortheright-handside.Oneofthereasonstoseparatedifferentitemsondifferenty-axissidesisthatmaybeyouareplottingonthesamegraphitemsthathaveabsolutevaluestogetherwithitemsthatexpressapercentilevalue.Inthiscase,itmakessensetoshowtheabsolutescaleononesideandthepercentileoneontheothersideofthegraph.
Anotherreasonmightbethatyouareplottingtogetheritemsthatwillshow,onaverage,verybigorverysmallvalues,andyoucanpredictaheadoftimetheonesthatwillgravitatetowardsthebottomofthescale,andtheonesthatwillmakethescalegoupwithbigvalues.Inthatcase,youmightwanttoseparatethetwo;otherwise,theitemswithbigvalueswillmaketheotherslookveryflatandnotveryinformativeonthechart.Thisisthecaseillustratedintheprecedinggraph:wepredictedthatthetotalnumberofquerieswouldbemuchbigger(bydefinition)comparedtoalltheothers,sowemoveditsyaxistotheright-handside.Here’stheresultofthegraphwecreated:
www.it-ebooks.info
Whatwehaven’tshownhere,butyoucaneasilyimagine,isthataswithalmosteverythinginZabbix,youarenotlimitedtographingitemsfromthesamehost:youcanjustaseasilygraphthesameitemfromdifferenthosts,orevendifferentitemsfromdifferenthosts.Youmightbeinterested,forexample,intrackingnetworktrafficfromabunchofdifferentroutersandlookingathowthistrafficchangesintime,whichmachinesarethebusiestandwhen,whichonesarenotasbusyasyouexpectedcomparedtotheoveralltrafficyouhave,andsoon.Todothat,youcaneasilycreateagraphfollowingtheguidelinesabove,onlyselectingtherelevantnetworkinterfacesinboundandoutbounditemsfromthedifferentappliancesandputtingthemallonthesameitemlist.
YoucanuseZabbix’scustomgraphcreationfacilitiestoexploreyourdatainverymeaningfulwaysthatcanbehardtoachieveotherwise:don’tbefooledbythefactthatit’sallmainlytime-based(youcan’tputcustomvaluesonthexaxis).You’llsoonfindthattheabilitytocorrelatedifferentitemsfromdifferentsourcesisaverypowerfultoolforbothtroubleshootingandcapacityplanning.
AnotherpowerfultoolisZabbix’smappingfacility.We’llexploreafewinterestingaspectsofmapcreationandmaintenanceinthefollowingsection.
www.it-ebooks.info
Maps–aquicksetupforalargetopologyCreatingcomplexmapsisthekindofjobthatcantakealotoftime.Whiledoingapracticalexample,ifyouwouldliketodesignamapof20-30elements,itiseasytospendupto2hoursevenifyoualreadyknowthejob.
Tomanuallyproduceamap,youneedto:
AddalltheitemsonthemapMovetheitemsarounduntilyouseeanice-lookingdisposition
Everytimeyouneedtoaddinamaponehost,youneedtorepeatmanytimesthesamestepsasaforementioned,whichwillbecomeaboringandcomplextask.Currently,therearemanyopen-featurerequeststhatcanfacilitatethiskindoftask;unfortunately,theyhavebeenopenforalongtime,evenyears.
Theissuesyoucanfaceare:
Youcan’tmovemultipleelementsatthesametime,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-161Youcan’taddhostsinabulkway,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-163Youcan’tcloneanyexistingmapelement,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-51Whenyouareusingicons,youcan’tselectthemautomatically,soyouneedtochecktheirsizeandseewhethertheyfitonyourmap,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-1608
Forallthoseissues,weneedtofindadifferentwaytoautomatethislongandslowprocess.Clearly,thisisthekindoftaskthatneedstobeautomatedasmuchaspossible.
www.it-ebooks.info
Maps–automatingtheDOTcreationWhatismissinghereissomethingthatcanprocessourinformationandproduceasoutputsomethingusablebyZabbix.Toautomatethistask,thereisonelibrarythatcanhelpus—NetworkX—whichisavailableathttp://networkx.github.io/.
NetworkXisaPythonsoftwarelibrarytailor-madeforthecreation,manipulation,andstudyofdynamicnetworkstructures.
Inthisexample,weassumethatyou’reusingCiscoPrime,whichisavendor-specifictooltoexportadiscoveredtopology.
Anyway,thisconceptisstillvalidasherewearegoingtouseanexportfileobtained,whichisinCSV.ThiskindofCSVcanbeobtainedasanexportfrommanyothervendors’softwareandcanbeeasilyproducedfromanythird-partysoftware.
Thefilethatwearegoingtoparseisinthefollowingform:
IPaddress,Systemname,SysObjectID,Foundbymodules,Neighbors,Status
Asyoucansee,itcontainstheIPaddressofthedevicediscovered,thesystemname,theOIDofthesystem,themodulethatfoundthedevice,alistofalltheneighborsthatareconnectedtoit,anditendswiththestatus.
Thefollowingisanexampleofthelinethatweareexpectingtosee:
10.12.50.1,main.example.com,.1.3.6.1.4.1.9.1.896,System,"10.12.2.1,
10.12.2.2,10.12.3.1,10.12.4.1,10.12.5.1",Reachable
Wearemostlyinterestedinthefollowingfields:
IPaddressSystemnameSysObjectIDNeighbors
Then,whatwecandoiswritesomePythonlinesthatcanreadthisfile,identifyalltherequiredinformation,andwriteintheoutputaDOTfile.
Here,IamgoingtospendafewwordsabouttheDOTnotation,performinganexampleinordertoclarifyhowthisnotationisdone.
Firstofall,IwouldliketoexplainwhywearegoingtohaveaGraphvizDOTfile.
TheGraphvizDOTfileisreallyeasytoread,maintain,andupdate,andnevertheless,itcanbestoredinaCVSorSVN.
Somethingthatisreallyimportanttohaveisafilethatcanbequicklyusedtospotallthedifferencesbetweenversionsandiseasytomaintain.Also,weareconsideringusingitasitisastandardlanguageandagoodstartingpoint,onwhichwecantransformallouracquireddatafromallthedifferentversionsofexport.
Indeed,someothervendor-specificsoftwarecanexportthesamedatabutinadifferentform,soitisimportanttonormalizeallourdatainacommonlanguage.
www.it-ebooks.info
ThiscommonlanguagefilewillbethefiletousetopopulateourZabbixmap.
Thissection,asyouprobablyalreadyhaveunderstood,willbealargeusageoftheGraphviz’spackages.
TheeasiestwaytoinstallandmaintainGraphvizonRedHatEnterpriseLinuxistousethededicatedyumrepository.Tosetupyum,firstofall,youneedtodownloadthegraphviz-rhel.repofileandsaveit(asroot)in/etc/yum.repos.d/,asfollows:
#cd/etc/yum.repos.d
#wgethttp://www.graphviz.org/graphviz-rhel.repo
--2014-11-2702:52:17--http://www.graphviz.org/graphviz-rhel.repo
Resolvingwww.graphviz.org…204.178.9.49
Connectingtowww.graphviz.org|204.178.9.49|:80…connected.
HTTPrequestsent,awaitingresponse…200OK
Length:1138(1.1K)[text/plain]
Savingto:"graphviz-rhel.repo"
100%[======================================>]1,138--.-K/sin0s
2014-11-2702:52:17(134MB/s)-"graphviz-rhel.repo"saved[1138/1138]
#ls-lagraphviz-rhel.repo
-rw-r--r--.1rootroot1138Feb162012graphviz-rhel.repo
Then,youcanfinallylistalltheGraphvizpackagesasroot:
yumlistavailable'graphviz*'
Installthem,asfollows:
yuminstall'graphviz*'
Nowthatwe’veclarifiedthereasonwhywe’redoingthosesteps,itisimportanttowalkthroughtheDOTlanguage.TheDOTlanguageisalanguagemadetorepresentobjectsconnectedbetweeneachother.
Whileperformingapracticalexample,ifwewanttodefinetwoconnectednodeswiththeGraphvizDOTlanguage,wecandoasfollows:
graph{
A—B
}
Thisisaveryeasy-to-understandlanguage;wearenowrepresentingtwonodesconnectedtoeachother.
Toseethegraphicalresult,wecanuseasimplePythonprogramxdot.pyavailablefordownloadhere:
https://github.com/jrfonseca/xdot.py
Allyouhavetodoisdownloadtheprogram,writeafilewiththeGraphvizDOTcontentthatweshowedpreviously,andthenruntheprogram,asfollows:
xdot.pyexample.dot
www.it-ebooks.info
TheresultistheDOTexpressedtopologyvisualized,asfollows:
Usingthesamegrammar,wecandefinethreenodesconnected,asfollows:
graph{
A—B—C
}
Usingthesamexdot.pyusedpreviously,theresultisthefollowing:
Writingacoupleoflinesmore,wecanevenavoidusinglongnamesusingthefollowinggrammar:
graph{
//Wecancreatealiasestoavoidtouseverylongnamesonthedependency
definition
Andrea[hostname="andrea.dalle.vacche.example.com"]
Stefano[hostname="stefano.kewan.lee.example.com"]
router[label="Ournetworkrouter"zbximage="router"]
//nowit'stimetodefineconnectionsbetweenthenodes
//Thisnotationallowsformultipleedgesfrom"router"inonego
router—{AndreaStefano}
}
www.it-ebooks.info
Andtheresultisshownhere:
Foradetaileddocumentationofthisgrammar,pleaserefertotheofficialdocumentationavailableathttp://www.graphviz.org/content/dot-language.
Untilnow,we’vecoveredallthatisneededtoknowforoursmallapplication.
Now,wecancomebacktoourCSVfileweextractedfromCiscoPrime.
HereistheCSVofaverysimplenetwork,butitcanbeappliedonverycomplexnetworktopologies,aswell:
[root@localhostgraphs]#catmy_export.csv
IPAddress,SystemName,SysObjectID,FoundByModules,Neighbors,Status
10.12.20.1,main.example.com,.1.3.6.1.4.1.9.1.896,System,"10.12.2.1,
10.12.2.2,10.12.3.1,10.12.4.1,10.12.5.1",Reachable
10.12.2.1,cluster1.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.2.2,
192.168.99.1",Reachable
10.12.1.1,london.example.com,.1.3.6.1.4.1.9.1.503,System,"",Reachable
10.12.2.2,cluster2.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.2.1,
192.168.99.1",Reachable
10.12.3.1,switch1.example.com,.1.3.6.1.4.1.9.1.503,System,"192.168.99.1",Re
achable
10.12.4.1,4.example.com,.1.3.6.1.4.1.9.1.502,System,"192.168.99.1,
10.12.4.42,10.12.4.47,10.12.4.48,10.12.4.49",Reachable
10.12.4.45,4d.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable
10.12.4.46,4e.example.com,.1.3.6.1.4.1.9.1.502,System,"10.12.4.1",Reachable
10.12.4.47,4f.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable
10.12.4.48,4g.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable
10.12.5.1,5.example.com,.1.3.6.1.4.1.9.1.502,System,"192.168.99.1,
10.12.5.45,10.12.5.43,10.12.5.44,10.12.5.46,10.12.5.47,10.12.5.48,
10.12.6.1",Reachable
10.12.5.44,5c.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable
10.12.5.45,5d.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable
10.12.5.46,5e.example.com,.1.3.6.1.4.1.9.1.502,System,"10.12.5.1",Reachable
10.12.5.47,5f.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable
10.12.5.48,5g.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable
10.12.5.155,5i.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.5.1",Reachabl
e
10.12.6.1,6.example.com,.1.3.6.1.4.1.9.1.502,System,"10.12.6.45,
10.12.6.46,10.12.6.47,,10.12.5.1",Reachable
10.12.6.45,6d.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.6.1",Reachable
10.12.6.46,6e.example.com,.1.3.6.1.4.1.9.1.502,System,"10.12.6.1",Reachable
www.it-ebooks.info
10.12.6.47,6f.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.6.1",Reachable
Fromthisfile,weseethatalltherelationsbetweenneighborsarealreadycontainedintheCSV,andthatweonlyneedtoconvertthemintoDOTnotationusingthenodenotation.
Here,wecanstartcodingafewPythonlinestoproduceourdesiredoutput:
#FirstofallweneedtoimportcsvandNetworkx
importcsv
importnetworkxasnx
#Thenweneedtodefinewhoisourzabbixserverandsomeotherdetailto
properlyproducetheDOTfile
zabbix_service_ipaddr="192.168.1.100"
main_loop_ipaddr="10.12.20.1"
main_vlan_ipaddr="149.148.56.1"
#Nowwecanfinallycreateourgraph
G=nx.Graph()
#wecanopenourCSVfile
csv_reader=csv.DictReader(open('my_export.csv'),\
delimiter=",",\
fieldnames=("ipaddress","hostname","oid","dontcare","neighbors"))
#Skiptheheader
csv_reader.next()
forrowincsv_reader:
neighbor_list=row["neighbors"].split(",")
forneighborinneighbor_list:
#Removespaces
neighbor=neighbor.lstrip()
#Addneighbors,andherewe'vedecidedtoignoreisolatednodes
ifneighbor!="":
G.add_edge(row["ipaddress"],neighbor)
#Addadditionalinformationtonodesoredgeshere
G.node[row["ipaddress"]]["hostname"]=row["hostname"]
#CiscoPrimedoesn'texportallIPaddressesofadevice
#butonlythefirstforeachnetwork,Herewemergehostswith
#multipleIPaddresses
mapping={main_vlan_ipaddr:main_loop_ipaddr}
G=nx.relabel_nodes(G,mapping)
#Removeclusterconnectionnotneededinourmap
G.remove_edge("10.12.2.1","10.12.2.2")
#AddingconnectionbetweenZabbixserverandmainswitch
G.add_edge(zabbix_service_ipaddr,main_loop_ipaddr)
main_neigh_list=G.neighbors(main_loop_ipaddr)
#finallywriteoutourfile
nx.draw_graphviz(G)
nx.write_dot(G,"/tmp/total.dot")
Now,ifyourunthissmallsoftwareagainsttheCSVfilewehaveshownbeforeyouseeourDOTfilegeneratedon/tmp/total.dot.Now,itisinterestingtoseehowourDOTfile
www.it-ebooks.info
isrepresentedonXDot.Here,inthenextdiagram,weseetherepresentationofourDOTfile:
Now,allthatwehavetodoisproducethemapstartingfromtheDOTfilewejustgenerated.
www.it-ebooks.info
DraftingZabbixmapsfromDOTHavingarrivedatthispoint,wehaveourGraphvizDOTfilethatiswaitingtobeused.Asyoucanseefromthepreviousimage,thankstoGraphviz,wealreadyhaveaready-to-goimagetouse.Then,allweneedtodois:
1. ReadouttheDOTfile.2. GeneratethetopologyusingGraphviz.3. Acquireallthecoordinatesfromourtopologygenerated.4. UsepyzabbixtoconnecttoourZabbixserver.5. Generateourtopologyinafullyautomatedway.
It’snowtimetowritesomelinesofPython;thefollowingexampleissimilartosomethingpresentedbyVolkerFröhlich.Anyway,thecodeherehasbeenchangedandfixed(itdidnotworkwellwithZabbix2.4).
Asthefirstthing,weneedtoimporttheZabbixApiandnetworkXlibraries:
importnetworkxasnx
frompyzabbiximportZabbixAPI
Then,wecandefinetheGraphvizDOTfiletouseasasource;agoodexampleistheonewejustgenerated:
dot_file="/tmp/total.dot"
Inthenextfewlines,wedefineourusername,password,mapdimension,andrelativemapname:
username="Admin"
password="zabbix"
width=800
height=600
mapname="my_network"
Whatfollowsisastaticmaptodefinetheelementtype:
ELEMENT_TYPE_HOST=0
ELEMENT_TYPE_MAP=1
ELEMENT_TYPE_TRIGGER=2
ELEMENT_TYPE_HOSTGROUP=3
ELEMENT_TYPE_IMAGE=4
ADVANCED_LABELS=1
LABEL_TYPE_LABEL=0
Then,wecandefinetheiconstouseandtherelativecolorcode:
icons={
"router":23,
"cloud":26,
"desktop":27,
"laptop":28,
"server":29,
"sat":30,
www.it-ebooks.info
"tux":31,
"default":40,
}
colors={
"purple":"FF00FF",
"green":"00FF00",
"default":"00FF00",
}
Now,wedefinesomefunctionsthatwecanreuse.Thefirstoneistomanagethelogin,andthesecondoneistodefineahostlookup,asfollows:
defapi_connect():
zapi=ZabbixAPI("http://127.0.0.1/zabbix/")
zapi.login(username,password)
returnzapi
defhost_lookup(hostname):
hostid=zapi.host.get({"filter":{"host":hostname}})
ifhostid:
returnstr(hostid[0]['hostid'])
Thenextthingtodo,isreadourDOTfileandstartconvertingitintoagraph:
G=nx.read_dot(dot_file)
Then,wecanfinallyopenourgraph,asfollows:
pos=nx.graphviz_layout(G)
NoteHere,youcanselectyourpreferredalgorithm.Graphvizsupportsmanydifferentkindsoflayout,andthenyoucanchangethelookandfeelofyourmapasyouprefer.FormoreinformationaboutGraphviz,pleasechecktheofficialdocumentationavailableathttp://www.graphviz.org/.
Then,asthegraphisalreadygenerated,thenextthingtodoisfindthemaximumcoordinatesofthelayout.Thiswillenableustoscalebetterourpredefinedmapoutputsize.
positionlist=list(pos.values())
maxpos=map(max,zip(*positionlist))
forhost,coordinatesinpos.iteritems():
pos[host]=[int(coordinates[0]*width/maxpos[0]*0.95-
coordinates[0]*0.1),int((height-
coordinates[1]*height/maxpos[1])*0.95+coordinates[1]*0.1)]
nx.set_node_attributes(G,'coordinates',pos)
NoteGraphvizandZabbixusetwodifferentdataorigins:Graphvizstartsfromthebottom-leftcorner,andZabbixworksstartingfromthetop-leftcorner.
Then,weneedtoretrievetheselementidsastheyarerequiredforlinksandevenforthenodedatacoordinates,asfollows:
www.it-ebooks.info
selementids=dict(enumerate(G.nodes_iter(),start=1))
selementids=dict((v,k)fork,vinselementids.iteritems())
nx.set_node_attributes(G,'selementid',selementids)
nx.set_node_attributes(G,'selementid',selementids)
Now,wedefinethemaponZabbix,thename,andtherelativemapsize:
map_params={
"name":mapname,
"label_type":0,
"width":width,
"height":height
}
element_params=[]
link_params=[]
Finally,wecanconnecttoourZabbixserver:
zapi=api_connect()
Then,prepareallthenodeinformationandthecoordinatesandthensettheicontouse,asfollows:
fornode,datainG.nodes_iter(data=True):
#Genericpart
map_element={}
map_element.update({
"selementid":data['selementid'],
"x":data['coordinates'][0],
"y":data['coordinates'][1],
"use_iconmap":0,
})
Checkwhetherwehavethehostname,asfollows:
if"hostname"indata:
map_element.update({
"elementtype":ELEMENT_TYPE_HOST,
"elementid":host_lookup(data['hostname'].strip('"')),
"iconid_off":icons['server'],
})
else:
map_element.update({
"elementtype":ELEMENT_TYPE_IMAGE,
"elementid":0,
})
Wesetlabelsforimages,asfollows:
if"label"indata:
map_element.update({
"label":data['label'].strip('"')
})
if"zbximage"indata:
map_element.update({
"iconid_off":icons[data['zbximage'].strip('"')],
})
www.it-ebooks.info
elif"hostname"notindataand"zbximage"notindata:
map_element.update({
"iconid_off":icons['default'],
})
element_params.append(map_element)
Now,weneedtoscanalltheedgestocreatetheelementlinksbasedontheelementweidentified,asfollows:
nodenum=nx.get_node_attributes(G,'selementid')
fornodea,nodeb,datainG.edges_iter(data=True):
link={}
link.update({
"selementid1":nodenum[nodea],
"selementid2":nerodenum[nodeb],
})
if"color"indata:
color=colors[data['color'].strip('"')]
link.update({
"color":color
})
else:
link.update({
"color":colors['default']
})
if"label"indata:
label=data['label'].strip('"')
link.update({
"label":label,
})
link_params.append(link)
#Jointhepreparedinformation
map_params["selements"]=element_params
map_params["links"]=link_params
Now,wehavepopulatedallmap_params,andnowweneedtocallZabbix’sAPIwiththisdata:
map=zapi.map.create(map_params)
Theprogramisnowcomplete,andwecanletitrun!Inareal-worldcase,thetimespenttodesignatopologyofmorethan2,500hostsisonly2–3seconds!
Wecantestthesoftwarehere,proposedagainsttheDOTfilewegeneratedbefore:
[root@localhost]#time./Generate_MyMap.py
real0m0.005s
user0m0.002s
sys0m0.003s
Asyoucansee,oursoftwareisreallyquick…butlet’scheckwhathasbeengenerated.In
www.it-ebooks.info
thenextscreenshot,youcanseethemapthatisgeneratedautomaticallyin0.005seconds:
www.it-ebooks.info
PuttingeverythingtogetherwithscreensUnlikeanyotherZabbixfeaturewedescribedinthischapter,screensdon’tactuallygiveyouneworimprovedinformationaboutyourmonitoreddata.PrettymuchanythingthatyoucandecidetoputonascreencanbefoundsomewhereelseinZabbix.
Frommapsandgraphs,totriggerstatusanditemdata,allofthisandmorecanbeeasilyfoundbyexploringtheMonitoringtabofthewebfrontend.
ButthepointofgatheringexistingdataonaZabbixscreenispreciselythatyoubringtogetherrelateddata,ordifferentviewsofthesamedatasothatyoudon’thavetolookforitaroundthefrontend,andsothatyoucanhaveagoodoverviewofthestatusofyoursystemsandseeataglancewhetherthereareanyproblemswithinyourinfrastructure.
Whenyoucreateascreen(Configuration|Screens|Createscreen),yougiveitanameandastartingnumberofrowsandcolumns.Don’tworrytoomuchabouthowmanyrowsandcolumnsyouassigntoascreenasyouwillbeabletochangethemduringscreenconfiguration.
Onceyouhavethescreencreated,youcangoaheadandconfigureitbyselectingitsnameinConfiguration|Screens.
Ascreenisbasicallyatablewithrowsandcolumnsthatidentifiescells.Everycellcancontaindifferenttypesofdata:
Celltype Description
Actionlog ThisshowsalogofthelatestactionsexecutedbyZabbix.Youcanconfigurehowmanyactionsyouwanttoseeinthecell.
Clock Thisshowsananalogclockwiththecurrenttime.
Dataoverview Thisshowsthelatestitemdataforaspecificgroupofhosts.
Graph Thisshowsanexistingcustomgraph.
Graphprototype Thisshowsacustomgraphcreatedfromalow-leveldiscoveryruleprototype.
Historyofevents
Thisshowsalogofthelatestevents(thesedon’tnecessarilyleadtoactions).Youcanconfigurehowmanyeventsyouwanttoseeinthecell.
Hostgroupissues Thisshowsthecurrentissuesforaspecifichostgroup.
Hostissues Thisshowsthecurrentissuesforaspecifichost.
Host’sinfo Thisshowsasummaryofhostavailabilityforaspecificgroup,suchastheoneyoufindinMonitoring|Overview.
Map Thisshowsanexistingmap.
www.it-ebooks.info
PlaintextThisshowstheplaintexthistoryofaspecificitemtogetherwiththetimestampforeachmeasurement.Youcanconfigurehowmanyentriesyouwanttoseeinthecell.
Screen Thisshowsanexistingscreen.Yes,youcanembedascreenintoanotherscreenifyouwant.
Serverinfo ThisshowsasummaryofthemonitoringstatusfortheZabbixserver,suchasDBconnectivity,numberofhosts,itemsandtriggers,newvaluespersecond,andsoon.
Simplegraph Thisshowsthegraphforasingleitem,suchastheonesyoucanseeinLatestdatawithoutcreatingacustomgraph.
Simplegraphprototype
Thisislikeasimplegraph,butisforitemscreatedautomaticallyfromalow-leveldiscoveryruleprototype.
Systemstatus Thisshowsasummaryofthecurrentissues,dividedintohostgroupsandseverity.
Triggerinformation
Thisshowsasummaryoftriggerscurrentlyinaproblemstate,dividedbyseverity.Youhavetospecifyahostgroup.
Triggeroverview Thisshowseverytriggerstatusforeveryhostinaspecifichostgroup(andoptionally,application).
URL Thisshowsthecontentofanarbitrarywebpage,givenitsURL.
Everycellisalsoindependentfromtheothers:youcanbringtogetherdatabelongingtothesamehostaswellasbelongingtodifferenthostsandhosts’groups,dependingonhowyouwanttoorganizeyourscreen.
Finally,foreverycell,youcanspecifyhowmanyrowsandcolumnsitshouldspan,andforgraphiccelltypes(maps,graphs,andsoon),youcanalsodefinehowmuchspacetheyshouldtakebyspecifyingthewidthandheightinpixels.
Allthisflexibilityiscertainlypowerfulbutcanbeabitoverwhelming,soherearesomegeneralguidelinesthatyoucanrefertowhenyoucreateyourownscreens.
Averyusefultypeofscreenbringstogetherdatafromasinglehostsothatyoucanseeataglanceitsoverallperformance.You’lltypicallywanttoseesomegraphsinascreenlikethis,suchasnetworkandCPUperformance,diskusage,andanyapplication-specificgraphoritemsummaryyoumightneed,suchasdatabaseperformancegraphs,applicationserverstatistics,andsoon.
Inthefollowingexample,we’vekeptthingssimpleduetospaceconstraints,butyoucanseehowevenfourgraphscanproveusefulwhenputtogetherthisway:
www.it-ebooks.info
Aninterestingfeatureofscreencellsisthatyoucanmakethecontentdynamicbyflaggingtheaptlynamedcheckbox.Dynamiccellswillreferthesametypeofcontenttodifferenthostsdependingonthecontext.
Thismeansthatyoucancreateascreenatthetemplatelevel,flagallcellsasdynamic,andjustlikethat,everyhostinheritingthetemplatewillalsoinheritapersonalizedscreen,withallgraphsandtablesreferencingtheaforesaidhost.Thisway,youwon’thavetomanuallycreateaspecificscreenforeveryhost.
Inanothertypeofscreen,youmightwanttofocusongrouptriggersandissues.Inthiskindofscreen,atypicalcell’scontentswillbesomemaps,withhostsandlinksthatchangecolorbasedontriggerstatus,sometriggerinformationandtriggeroverviewcells,andpossiblyalogofthelatesteventsandactions.
Finally,youmightwanttocreatespecificscreensthatbringtogetherhistoricaldatafromdifferentitems,suchasapplication-specificlogfiles,outputfromexternalcommands,suchasNmap,Windowsupdatestatusforahost,andsoon.Asusual,thesky’sthelimithere.
TipKeepinmindthattheprecedingscreentypesaremerelyexamplesthatbarelyscratchthesurfaceofwhat’spossiblewithZabbix’sscreen.Youarebynomeanslimitedtothesetypes;onthecontrary,youareencouragedtomixandmatchthedifferentcellstosuityourownneeds.Don’tletusstopyoufromcreatingawesomescreens!
Onceyouhavecreatedafewscreens,thenextlogicalstepistofindawaytobringthemtogetherinanorganizedway.Slideshowsservethispurposeinaninterestingandusefulway.YoucancreateaslideshowbygoingtoConfiguration|Slideshowsandclicking
www.it-ebooks.info
onCreateslideshow.Thecreationformisprettyself-explanatory:
Muchlikeaddingitemstoacustomgraph,byclickingontheAddlinkatthebottomoftheSlideslist,youcanaddexistingscreenstotheslideshow,andyoucanreorderthembydragginganddroppingthebluearrowsnearthescreennameinthelist.Theresultwillbe,quitepredictably,aslideshowofallthescreensyouhaveputinthelist.Itwillrunoverandovercyclingthroughalltheelements.Eachslidewillhavethefocusforthenumberofsecondsequaltothedefaultdelayifyoudon’tspecifyanythingintheslide’sDelayfield.
Slideshowsareveryusefulwhenshownonabigscreeninadatacenter,butyouneedtobecarefulwhencreatingscreensthatyouknowwillendupinaslideshow.Slidesdon’tscrollvertically,soifascreenisbiggerthanthebrowserwindowusedtoshowtheslides,you’llneverbeabletoseesomeofthedata.Apossibleworkaroundistocreatescreensthatwilltakeupthewholewindowsize,butnothingmore.Thisway,you’llbesurethatallrelevantdatawillalwaysshowupontheslideshowthatyouplayonthatbigscreenyouputonthewallformonitoringpurposes.
Anotherworkaroundistomakesurethatforeachscreenbiggerthanthewindowsize,youputallimportantdataatthetopofthescreen.Thisway,someofthescreen’sdatawillshowupontheslides,whileyou’llstillbeabletoaccessallofitwhenaccessingthescreenonitsownandnotaspartoftheslideshow.
www.it-ebooks.info
SummaryInthischapter,youexploredZabbix’svisualizationfeaturesandlearnedhowtousethemtogetthemostoutofyourmonitoringdata.Sometimes,thevalueofameasurementdoesn’tlieintheeventsandactionsthatitcantrigger,butinitscorrelationwithothermeasurements,bothintime(graphs)andinstantly(maps).Thisisespeciallytruewithnetworkmonitoring,wheretheabilitytopredictthefutureneedsofanetwork,andadapttothem,isjustasimportantasactingoncontingentissues.
WehavereachedtheendofourbriefjourneythroughZabbix’sconfigurationanduse.Now,youshouldbeabletocorrectlysizeaZabbixinstallationbasedonyouenvironment;findthebestandmostappropriatetoolsandprotocolstomonitoryourdata;automatedevicediscoveryandmonitoringasmuchaspossible(andwhennottoautomateit);andmovebeyondactionsandtriggersandvisualizeallyourdatainmeaningfulways.
Withalltheseskillsunderyourbelt,weareconfidentthatyou’llbeabletoadaptapowerfulandflexibletoollikeZabbixtoyourownnetworkandnotbeconfinedtodefaulttemplatesthatmay,ormaynot,reflectyouractualmonitoringneeds.
Monitoringacomputernetworkisoftenalsoadiscoveryjourney,whereyoucangainunexpectedwisdomfromapparentlydryanduninspiringdata,suchasSNMPvaluesandserverlogs.Withthisshortbook,wehopewehaveshownyouhowZabbixcanbeanexcellentmeanstogainsuchwisdomifyouarewillingtoplaywithitforawhileandputtogooduseallitspowerfulfeatures.
www.it-ebooks.info
MySQLpartitioningHereareallthestoredproceduresyouneedtocreatetoproperlyhandledatabasepartitioningwithMySQL.
YouneedtocreatealloftheminyourZabbixdatabase.
Notethatalltheproceduresdescribedherearealsoavailableathttps://github.com/smartmarmot/zabbix_network_monitoring/tree/master/Chapter1.
www.it-ebooks.info
Thepartition_maintenanceprocedureThisisthemostimportantprocedure,whichwillmanagealltheotherstoredproceduresinvolvedinthecreation/dropandverificationofpartitions,asfollows:
DELIMITER$$
CREATEPROCEDURE`partition_maintenance`(SCHEMA_NAMEVARCHAR(32),
TABLE_NAMEVARCHAR(32),KEEP_DATA_DAYSINT,HOURLY_INTERVALINT,
CREATE_NEXT_INTERVALSINT)
BEGIN
DECLAREOLDER_THAN_PARTITION_DATEVARCHAR(16);
DECLAREPARTITION_NAMEVARCHAR(16);
DECLARELESS_THAN_TIMESTAMPINT;
DECLARECUR_TIMEINT;
CALLpartition_verify(SCHEMA_NAME,TABLE_NAME,HOURLY_INTERVAL);
SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(NOW(),'%Y-%m-%d
00:00:00'));
IFDATE(NOW())='2014-04-01'THEN
SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD(NOW(),
INTERVAL1DAY),'%Y-%m-%d00:00:00'));
ENDIF;
SET@__interval=1;
create_loop:LOOP
IF@__interval>CREATE_NEXT_INTERVALSTHEN
LEAVEcreate_loop;
ENDIF;
SETLESS_THAN_TIMESTAMP=CUR_TIME+(HOURLY_INTERVAL*
@__interval*3600);
SETPARTITION_NAME=FROM_UNIXTIME(CUR_TIME+
HOURLY_INTERVAL*(@__interval-1)*3600,'p%Y%m%d%H00');
CALLpartition_create(SCHEMA_NAME,TABLE_NAME,
PARTITION_NAME,LESS_THAN_TIMESTAMP);
SET@__interval=@__interval+1;
ENDLOOP;
SETOLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(),INTERVAL
KEEP_DATA_DAYSDAY),'%Y%m%d0000');
CALLpartition_drop(SCHEMA_NAME,TABLE_NAME,
OLDER_THAN_PARTITION_DATE);
END$$
DELIMITER;
Thisstoredprocedurewillbethecoreofourhousekeeping.Itwillbecalledwiththefollowingsyntax:
CALLpartition_maintenance('<zabbix_db_name>','<table_name>',
<days_to_keep_data>,<hourly_interval>,<num_future_intervals_to_create>)
www.it-ebooks.info
Thepartition_createprocedureThisprocedureisresponsibleforcreatingnewpartitionsacrossyourschema.Whatfollowshereistheprocedureitself:
DELIMITER$$
CREATEPROCEDURE`partition_create`(SCHEMANAMEVARCHAR(64),TABLENAME
VARCHAR(64),PARTITIONNAMEVARCHAR(64),CLOCKINT)
BEGIN
/*
SCHEMANAME=TheDBschemainwhichtomakechanges
TABLENAME=Thetablewithpartitionstopotentiallydelete
PARTITIONNAME=Thenameofthepartitiontocreate
*/
/*
Verifythatthepartitiondoesnotalreadyexist
*/
DECLARERETROWSINT;
SELECTCOUNT(1)INTORETROWS
FROMinformation_schema.partitions
WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAMEAND
partition_name=PARTITIONNAME;
IFRETROWS=0THEN
/*
1.Printamessageindicatingthatapartitionwas
created.
2.CreatetheSQLtocreatethepartition.
3.ExecutetheSQLfrom#2.
*/
SELECTCONCAT("partition_create(",SCHEMANAME,",",
TABLENAME,",",PARTITIONNAME,",",CLOCK,")")ASmsg;
SET@SQL=CONCAT('ALTERTABLE',SCHEMANAME,'.',
TABLENAME,'ADDPARTITION(PARTITION',PARTITIONNAME,'VALUESLESSTHAN
(',CLOCK,'));');
PREPARESTMTFROM@SQL;
EXECUTESTMT;
DEALLOCATEPREPARESTMT;
ENDIF;
END$$
DELIMITER;
www.it-ebooks.info
Thepartition_verifyprocedureThispartitionisresponsibleforverifyingwhetherapartitionisalreadypresent,andifitisn’t,partition_verifywillcreatethem,asfollows:
DELIMITER$$
CREATEPROCEDURE`partition_verify`(SCHEMANAMEVARCHAR(64),TABLENAME
VARCHAR(64),HOURLYINTERVALINT(11))
BEGIN
DECLAREPARTITION_NAMEVARCHAR(16);
DECLARERETROWSINT(11);
DECLAREFUTURE_TIMESTAMPTIMESTAMP;
/*
*Checkifanypartitionsexistforthegiven
SCHEMANAME.TABLENAME.
*/
SELECTCOUNT(1)INTORETROWS
FROMinformation_schema.partitions
WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAMEAND
partition_nameISNULL;
/*
*Ifpartitionsdonotexist,goaheadandpartitionthetable
*/
IFRETROWS=1THEN
/*
*Takethecurrentdateat00:00:00andaddHOURLYINTERVAL
toit.Thisisthetimestampbelowwhichwewillstorevalues.
*Webeginpartitioningbasedonthebeginningofaday.
Thisisbecausewedon'twanttogeneratearandompartition
*thatwon'tnecessarilyfallinlinewiththedesired
partitionnaming(ie:ifthehourintervalis24hours,wecould
*endupcreatingapartitionnownamed"p201403270600"
whenallotherpartitionswillbelike"p201403280000").
*/
SETFUTURE_TIMESTAMP=TIMESTAMPADD(HOUR,HOURLYINTERVAL,
CONCAT(CURDATE(),"",'00:00:00'));
SETPARTITION_NAME=DATE_FORMAT(CURDATE(),'p%Y%m%d%H00');
—Createthepartitioningquery
SET@__PARTITION_SQL=CONCAT("ALTERTABLE",SCHEMANAME,
".",TABLENAME,"PARTITIONBYRANGE(`clock`)");
SET@__PARTITION_SQL=CONCAT(@__PARTITION_SQL,"(PARTITION
",PARTITION_NAME,"VALUESLESSTHAN(",UNIX_TIMESTAMP(FUTURE_TIMESTAMP),
"));");
—Runthepartitioningquery
PREPARESTMTFROM@__PARTITION_SQL;
EXECUTESTMT;
DEALLOCATEPREPARESTMT;
ENDIF;
END$$
DELIMITER;
www.it-ebooks.info
Thepartition_dropprocedureThisstoredprocedureisresponsiblefordroppingthepartitionsolderthanagivenperiod,asfollows:
DELIMITER$$
CREATEPROCEDURE`partition_drop`(SCHEMANAMEVARCHAR(64),TABLENAME
VARCHAR(64),DELETE_BELOW_PARTITION_DATEBIGINT)
BEGIN
/*
SCHEMANAME=TheDBschemainwhichtomakechanges
TABLENAME=Thetablewithpartitionstopotentiallydelete
DELETE_BELOW_PARTITION_DATE=Deleteanypartitionswithnames
thataredatesolderthanthisone(yyyy-mm-dd)
*/
DECLAREdoneINTDEFAULTFALSE;
DECLAREdrop_part_nameVARCHAR(16);
/*
Getalistofallthepartitionsthatareolderthanthedate
inDELETE_BELOW_PARTITION_DATE.Allpartitionsareprefixed
with
a"p",souseSUBSTRINGTOgetridofthatcharacter.
*/
DECLAREmyCursorCURSORFOR
SELECTpartition_name
FROMinformation_schema.partitions
WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAME
ANDCAST(SUBSTRING(partition_nameFROM2)ASUNSIGNED)<
DELETE_BELOW_PARTITION_DATE;
DECLARECONTINUEHANDLERFORNOTFOUNDSETdone=TRUE;
/*
Createthebasicsforwhenweneedtodropthepartition.Also,
create
@drop_partitionstoholdacomma-delimitedlistofall
partitionsthat
shouldbedeleted.
*/
SET@alter_header=CONCAT("ALTERTABLE",SCHEMANAME,".",
TABLENAME,"DROPPARTITION");
SET@drop_partitions="";
/*
Startloopingthroughallthepartitionsthataretooold.
*/
OPENmyCursor;
read_loop:LOOP
FETCHmyCursorINTOdrop_part_name;
IFdoneTHEN
LEAVEread_loop;
ENDIF;
SET@drop_partitions=IF(@drop_partitions="",
drop_part_name,CONCAT(@drop_partitions,",",drop_part_name));
ENDLOOP;
www.it-ebooks.info
IF@drop_partitions!=""THEN
/*
1.BuildtheSQLtodropallthenecessarypartitions.
2.RuntheSQLtodropthepartitions.
3.Printoutthetablepartitionsthatweredeleted.
*/
SET@full_sql=CONCAT(@alter_header,@drop_partitions,
";");
PREPARESTMTFROM@full_sql;
EXECUTESTMT;
DEALLOCATEPREPARESTMT;
SELECTCONCAT(SCHEMANAME,".",TABLENAME)AS`table`,
@drop_partitionsAS`partitions_deleted`;
ELSE
/*
Nopartitionsarebeingdeleted,soprintout"N/A"(Not
applicable)toindicate
thatnochangesweremade.
*/
SELECTCONCAT(SCHEMANAME,".",TABLENAME)AS`table`,"N/A"
AS`partitions_deleted`;
ENDIF;
END$$
DELIMITER;
www.it-ebooks.info
Thepartition_maintenance_allprocedureThisprocedurecallsthepartition_maintenanceprocedureforeachhistory/trendtable.Pleasenotethatforallthehistorytables,weareapplyingthesameintervals,whichare730daysoftrenddataand28daysofhistorydata.Here’showthisprocedureworks:
DELIMITER$$
CREATEPROCEDURE`partition_maintenance_all`(SCHEMA_NAMEVARCHAR(32))
BEGIN
CALLpartition_maintenance(SCHEMA_NAME,'history',28,24,
14);
CALLpartition_maintenance(SCHEMA_NAME,'history_log',28,
24,14);
CALLpartition_maintenance(SCHEMA_NAME,'history_str',28,
24,14);
CALLpartition_maintenance(SCHEMA_NAME,'history_text',28,
24,14);
CALLpartition_maintenance(SCHEMA_NAME,'history_uint',28,
24,14);
CALLpartition_maintenance(SCHEMA_NAME,'trends',730,24,
14);
CALLpartition_maintenance(SCHEMA_NAME,'trends_uint',730,
24,14);
END$$
DELIMITER;
www.it-ebooks.info
HousekeepingconfigurationAsperourexample,thehousekeepingneedstobeconfigured,asshowninthefollowingscreenshot,withahistorydatastorageperiodof730daysandatrenddatastorageperiodof28days.Here,youcanchangethosevaluesbearinginmindthatyoualsoneedtochangetheparameterpassedtothestoredprocedures.
Tochangethehousekeepingsettinginthewebinterface,yousimplyneedtogotoAdministration|General|Housekeeping(fromthedrop-downlist),andhereistheconfiguration:
www.it-ebooks.info
SquidmetricscriptHere,youcanfindthescriptwediscussedinChapter3,MonitoringYourNetworkServices,andcreatethescriptintheusuallocation,thatis,at/home/zabbix/bin/squidcheck.sh.
Createthescriptwiththefollowingcontent:
catsquidcheck.sh
#!/bin/bash
VERSION="1.0"
functionusage()
{
echo"squidcheckversion:$VERSION"
echo"usage:"
echo"$0http_requests-NumberofHTTPrequestsreceived"
echo"$0clients-Numberofclientsaccessing
cache"
echo"$0icp_received-NumberofICPmessagesreceived"
echo"$0icp_sent-NumberofICPmessagessent"
echo"$0icp_queued-NumberofqueuedICPreplies"
echo"$0htcp_received-NumberofHTCPmessagesreceived"
echo"$0htcp_sent-NumberofHTCPmessagessent"
echo"$0req_fail_ratio-Requestfailureratio"
echo"$0avg_http_req_per_min-AverageHTTPrequestsperminute
sincestart"
echo"$0avg_icp_msg_per_min-AverageICPmessagesperminute
sincestart"
echo"$0request_hit_ratio-RequestHitRatios"
echo"$0byte_hit_ratio_5-ByteHitRatio5mins"
echo"$0byte_hit_ratio_60-ByteHitRatio60mins"
echo"$0request_mem_hit_ratio_5-RequestMemoryHitRatios5mins"
echo"$0request_mem_hit_ratio_60-RequestMemoryHitRatios60
mins"
echo"$0request_disk_hit_ratio_5-RequestDiskHitRatios5mins"
echo"$0request_disk_hit_ratio_60-RequestDiskHitRatios60mins"
echo"$0servicetime_httpreq-HTTPRequests(All)"
echo"$0process_mem-ProcessDataSegmentSizevia
sbrk"
echo"$0cpu_usage-CPUUsage"
echo"$0cache_size_disk-StorageSwapsize"
echo"$0cache_size_mem-StorageMemsize"
echo"$0mean_obj_size-MeanObjectSize"
echo"$0filedescr_max-Maximumnumberoffile
descriptors"
echo"$0filedescr_avail-Availablenumberoffile
descriptors"
}
########
#Main#
########
www.it-ebooks.info
if[[$#!=1]];then
#NoParameter
usage
exit0
fi
case$1in
"http_requests")
value="`squidclientmgr:info|grep'NumberofHTTPrequests
received:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"clients")
value="`squidclientmgr:info|grep'Numberofclientsaccessing
cache:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"icp_received")
value="`squidclientmgr:info|grep'NumberofICPmessages
received:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"icp_sent")
value="`squidclientmgr:info|grep'NumberofICPmessages
sent:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"icp_queued")
value="`squidclientmgr:info|grep'NumberofqueuedICP
replies:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"htcp_received")
value="`squidclientmgr:info|grep'NumberofHTCPmessages
received:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"htcp_sent")
value="`squidclientmgr:info|grep'NumberofHTCPmessages
sent:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"req_fail_ratio")
value="`squidclientmgr:info|grep'Requestfailureratio:'|cut-
d':'-f2|tr-d'\t'`"
rval=$?;;
"avg_http_req_per_min")
value="`squidclientmgr:info|grep'AverageHTTPrequestsperminute
sincestart:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"avg_icp_msg_per_min")
value="`squidclientmgr:info|grep'AverageICPmessagesperminute
sincestart:'|cut-d':'-f2|tr-d'\t'`"
rval=$?;;
"request_hit_ratio")
value="`squidclientmgr:info|grep'RequestHitRatios:'|cut-d':'-
f3|cut-d','-f1|tr-d'%'`"
rval=$?;;
"byte_hit_ratio_5")
value="`squidclientmgr:info|grep'Hitsas%ofbytessent:'|awk
-F'[:,%]''{print$10}'|tr-d'\t'`"
rval=$?;;
"byte_hit_ratio_60")
value="`squidclientmgr:info|grep'Hitsas%ofbytessent:'|awk
www.it-ebooks.info
-F'[:,%]''{print$15}'|tr-d'\t'`"
rval=$?;;
"request_mem_hit_ratio_5")
value="`squidclientmgr:info|grep'Hitsas%ofallrequests:'|
awk-F'[:,%]''{print$10}'|tr-d'\t'`"
rval=$?;;
"request_mem_hit_ratio_60")
value="`squidclientmgr:info|grep'Hitsas%ofallrequests:'|
awk-F'[:,%]''{print$15}'|tr-d'\t'`"
rval=$?;;
"request_disk_hit_ratio_5")
value="`squidclientmgr:info|grep'Diskhitsas%ofhit
requests:'|awk-F'[:,%]''{print$11}'|tr-d'\t'`"
rval=$?;;
"request_disk_hit_ratio_60")
value="`squidclientmgr:info|grep'Diskhitsas%ofhit
requests:'|awk-F'[:,%]''{print$16}'|tr-d'\t'`"
rval=$?;;
"servicetime_httpreq")
value="`squidclientmgr:info|grep'HTTPRequests(All):'|cut-d':'
-f2|tr-s''|awk'{print$1}'`"
rval=$?;;
"process_mem")
value="`squidclientmgr:info|grep'ProcessDataSegmentSizevia
sbrk'|cut-d':'-f2|awk'{print$1}'`"
rval=$?;;
"cpu_usage")
value="`squidclientmgr:info|grep'CPUUsage:'|cut-d':'-f2|tr-d
'%'|tr-d'\t'`"
rval=$?;;
"cache_size_disk")
value="`squidclientmgr:info|grep'StorageSwapsize:'|cut-d':'-
f2|awk'{print$1}'`"
rval=$?;;
"cache_size_mem")
value="`squidclientmgr:info|grep'StorageMemsize:'|cut-d':'-
f2|awk'{print$1}'`"
rval=$?;;
"mean_obj_size")
value="`squidclientmgr:info|grep'MeanObjectSize:'|cut-d':'-
f2|awk'{print$1}'`"
rval=$?;;
"filedescr_max")
value="`squidclientmgr:info|grep'Maximumnumberoffile
descriptors:'|cut-d':'-f2|awk'{print$1}'`"
rval=$?;;
"filedescr_avail")
value="`squidclientmgr:info|grep'Availablenumberoffile
descriptors:'|cut-d':'-f2|awk'{print$1}'`"
rval=$?;;
*)
usage
exit1;;
esac
if["$rval"-eq0-a-z"$value"];then
www.it-ebooks.info
rval=1
fi
if["$rval"-ne0];then
echo"ZBX_NOTSUPPORTED"
fi
echo$value
www.it-ebooks.info
IndexA
actionconditionssection/FindinghoststheZabbixwayactiondefinitionsection/Definingactionconditionsactionoperationssection/FindinghoststheZabbixwayApache
modules/ApachemonitoringApachemonitoring
about/Apachemonitoringperforming/Apachemonitoring
architectures,Zabbixabout/Zabbixarchitectures
www.it-ebooks.info
Ccomplexmaps
issues/Maps–aquicksetupforalargetopologyCPULoadparameter/Apachemonitoringcustomgraphs
creating/Creatingcustomgraphs
www.it-ebooks.info
Ddatabase
installing/Installingadatabasesize,considering/Consideringthedatabasesizeitems/Consideringthedatabasesizerefreshrate/Consideringthedatabasesizespace/ConsideringthedatabasesizeMySQLpartitioning/MySQLpartitioning
dataflow,Zabbixabout/UnderstandingZabbixdataflow
datatypes,SNMPabout/GettingdatatypesrightURL/GettingdatatypesrightINTEGER/GettingdatatypesrightSTRING/GettingdatatypesrightOID/GettingdatatypesrightIpAddress/GettingdatatypesrightCounter32/GettingdatatypesrightGauge32/GettingdatatypesrightCounter64/GettingdatatypesrightTimeTicks/Gettingdatatypesright
digabout/DNS–responsetime
discoveryitemsabout/Low-leveldiscovery
discoveryrulesabout/Low-leveldiscovery
DNSmonitoringabout/MonitoringtheDNSperforming/MonitoringtheDNSresponsetime,monitoring/DNS–responsetimeDNSSECzonerollover,monitoring/DNSSEC–monitoringthezonerollover
DNSSECparametersabout/DNSSEC–monitoringthezonerollover
www.it-ebooks.info
Ggraph
putting,onscreen/Puttingeverythingtogetherwithscreens
www.it-ebooks.info
Hhostgroups
about/Hostsandhostgroupsroutersgroup/Hostsandhostgroupsswitchesgroup/Hostsandhostgroupssubnetgroup/Hostsandhostgroups
hostsabout/UnderstandingZabbixhostsinterfaces/Hostinterfacesinventory/Hostinventory
housekeepingconfigurationabout/Housekeepingconfiguration
www.it-ebooks.info
IICMPechochecks
about/Simplechecksinterfaces/HostinterfacesInternetProtocolFlowInformationeXport(IPFIX)/Gettingnetflowfromthedevicestothemonitoringserver
www.it-ebooks.info
Llow-leveldiscovery
about/Low-leveldiscoveryadvantage/Low-leveldiscoveryrules,creating/Low-leveldiscoveryrules,managing/Low-leveldiscovery
www.it-ebooks.info
Mmaps
complexmaps/Maps–aquicksetupforalargetopologyDOTcreation,automating/Maps–automatingtheDOTcreationdrafting,fromDOT/DraftingZabbixmapsfromDOTputting,onscreen/Puttingeverythingtogetherwithscreens
MIBsabout/FindingtherightOIDstomonitor
MySQLpartitioningabout/MySQLpartitioningbenefits/MySQLpartitioningstoredprocedures/MySQLpartitioningpartition_maintenanceprocedure/Thepartition_maintenanceprocedurepartition_createprocedure/Thepartition_createprocedurepartition_verifyprocedure/Thepartition_verifyprocedurepartition_dropprocedure/Thepartition_dropprocedurepartition_maintenance_allprocedure/Thepartition_maintenance_allprocedure
www.it-ebooks.info
Nnetflow
about/Gettingnetflowfromthedevicestothemonitoringserverdata,gettingintoZabbix/Gettingnetflowfromthedevicestothemonitoringserverdata,receivingonserver/Receivingnetflowdataonyourserver
networkdiscoveryhosts,finding/FindinghoststheZabbixwayactionconditions,defining/Definingactionconditionsactionoperations,selecting/Choosingactionoperationsremotecommands,executing/Remotecommands
networkinterfacesabout/Low-leveldiscovery
networkservicesDNS,monitoring/MonitoringtheDNSApache,monitoring/ApachemonitoringNTP,monitoring/NTPmonitoringSquid,monitoring/Squidmonitoring
NetworkXURL/Maps–automatingtheDOTcreationabout/Maps–automatingtheDOTcreation
Nfdumpabout/Receivingnetflowdataonyourservernfcapd/Receivingnetflowdataonyourservernfdump/ReceivingnetflowdataonyourserverURL,fornfdumppackage/Receivingnetflowdataonyourserver
Nmap/ChoosingactionoperationsNTPmonitoring
about/NTPmonitoringperforming/NTPmonitoring,NTP–whatarewemonitoring?Delay/NTP–whatarewemonitoring?Offset/NTP–whatarewemonitoring?Jitter/NTP–whatarewemonitoring?
www.it-ebooks.info
OOIDs
finding,formonitoring/FindingtherightOIDstomonitorabout/FindingtherightOIDstomonitormapping,toZabbixitems/MappingSNMPOIDstoZabbixitems
www.it-ebooks.info
Ppartition_createprocedure
about/Thepartition_createprocedurepartition_dropprocedure
about/Thepartition_dropprocedurepartition_maintenanceprocedure
about/Thepartition_maintenanceprocedurepartition_maintenance_allprocedure
about/Thepartition_maintenance_allprocedurepartition_verifyprocedure
about/Thepartition_verifyprocedurePerlmodules
about/DNSSEC–monitoringthezonerolloverproxiesdataflow,Zabbix
about/UnderstandingtheZabbixproxies’dataflowProxyConfigFrequency=parameter
about/UnderstandingtheZabbixproxies’dataflowProxyDataFrequency=parameter
about/UnderstandingtheZabbixproxies’dataflowpyzabbix
about/RemotecommandsURL/Remotecommands
www.it-ebooks.info
RReadingRequestparameter/ApachemonitoringReqPerSecparameter/Apachemonitoringrollstateplugin
about/DNSSEC–monitoringthezonerollover
www.it-ebooks.info
Sscreen
about/Puttingeverythingtogetherwithscreenscreating/Puttingeverythingtogetherwithscreensmaps,puttingon/Puttingeverythingtogetherwithscreensgraph,puttingon/Puttingeverythingtogetherwithscreens
SiegeURL/Apachemonitoring
simplechecksabout/SimplechecksIcmpping/SimplechecksIcmppingloss/SimplechecksIcmppingsec/SimplechecksNet.tcp.service/SimplechecksNet.tcp.service.perf/Simplechecksconfiguring/Simplechecks
slideshowcreating/Puttingeverythingtogetherwithscreens
SNMPabout/KeepingSNMPsimpledata,gettingintoZabbix/GettingSNMPdataintoZabbixOIDs,findingformonitoring/FindingtherightOIDstomonitorOIDs,mappingtoZabbixitems/MappingSNMPOIDstoZabbixitemsdatatypes/Gettingdatatypesrightnetflowdata,receivingonserver/Receivingnetflowdataonyourserverlogfile,monitoringwithZabbix/MonitoringalogfilewithZabbix
SNMPgetsabout/KeepingSNMPsimple
snmptrapdabout/Snmptrapd
SNMPtrapsabout/KeepingSNMPsimple,SNMPtrapssnmptrapd/Snmptrapdtransforming,intoZabbixitem/TransformingatrapintoaZabbixitemnetflow,gettingfromdevices/Gettingnetflowfromthedevicestothemonitoringserver
Squidabout/SquidmonitoringURL/Squidmonitoring
Squidmetricscriptabout/Squidmetricscript
Squidmonitoringperforming/Squidmonitoring
www.it-ebooks.info
StartProxyPollers=parameterabout/UnderstandingtheZabbixproxies’dataflow
www.it-ebooks.info
TTCP/IPconnectionchecks
about/Simplecheckstriggerinformationcell/Puttingeverythingtogetherwithscreenstriggeroverviewcell/Puttingeverythingtogetherwithscreens
www.it-ebooks.info
WWaitingForConnectionparameter/ApachemonitoringWebGUIinterface
installing/InstallingtheWebGUIinterface
www.it-ebooks.info
Xxdot.py
URL/Maps–automatingtheDOTcreationxml2
about/MonitoringtheDNS
www.it-ebooks.info
ZZabbix
architectures/Zabbixarchitecturesdataflow/UnderstandingZabbixdataflowproxiesdataflow/UnderstandingtheZabbixproxies’dataflowinstalling/InstallingZabbixdatabase,installing/Installingadatabasehosts/UnderstandingZabbixhostshostgroups/Hostsandhostgroups
Zabbixagentpackage,forLinuxOSURL/CreatingaZabbixagentpackagewithCheckInstall
Zabbixagentsabout/GoingbeyondZabbixagentssimplechecks/SimplechecksSNMP/KeepingSNMPsimpleSNMPtraps/SNMPtraps
ZabbixApacheUpdaterplugin/ApachemonitoringZabbixinstallation
about/InstallingZabbixinstalling,frompackages/InstallingfrompackagesZabbixagent,settingup/SettingupaZabbixagentZabbixagentpackage,creatingwithCheckInstall/CreatingaZabbixagentpackagewithCheckInstallserverconfiguration/Serverconfiguration
Zabbixproxyinstalling/InstallingaZabbixproxy
zapacheplugin/ApachemonitoringURL/Apachemonitoring
zonestatepluginabout/DNSSEC–monitoringthezonerollover
www.it-ebooks.info