zabbix network monitoring essentials - omid-online.com · table of contents zabbix network...

www.it-ebooks.info

ZabbixNetworkMonitoringEssentials

www.it-ebooks.info

TableofContents

Credits

AbouttheAuthors

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Errata

Piracy

Questions

1.InstallingaDistributedZabbixSetup

Zabbixarchitectures

UnderstandingZabbixdataflow

UnderstandingtheZabbixproxies’dataflow

InstallingZabbix

Installingfrompackages

SettingupaZabbixagent

CreatingaZabbixagentpackagewithCheckInstall

Serverconfiguration

Installingadatabase

www.it-ebooks.info

Consideringthedatabasesize

MySQLpartitioning

InstallingaZabbixproxy

InstallingtheWebGUIinterface

Summary

2.ActiveMonitoringofYourDevices

UnderstandingZabbixhosts

Hostsandhostgroups

Hostinterfaces

Hostinventory

GoingbeyondZabbixagents

Simplechecks

KeepingSNMPsimple

GettingSNMPdataintoZabbix

FindingtherightOIDstomonitor

MappingSNMPOIDstoZabbixitems

Gettingdatatypesright

SNMPtraps

Snmptrapd

TransformingatrapintoaZabbixitem

Gettingnetflowfromthedevicestothemonitoringserver

Receivingnetflowdataonyourserver

MonitoringalogfilewithZabbix

Summary

3.MonitoringYourNetworkServices

MonitoringtheDNS

DNS–responsetime

DNSSEC–monitoringthezonerollover

Apachemonitoring

NTPmonitoring

NTP–whatarewemonitoring?

www.it-ebooks.info

Squidmonitoring

Summary

4.DiscoveringYourNetwork

FindinghoststheZabbixway

Definingactionconditions

Choosingactionoperations

Remotecommands

Low-leveldiscovery

Summary

5.VisualizingYourTopologywithMapsandGraphs

Creatingcustomgraphs

Maps–aquicksetupforalargetopology

Maps–automatingtheDOTcreation

DraftingZabbixmapsfromDOT

Puttingeverythingtogetherwithscreens

Summary

A.PartitioningtheZabbixDatabase

MySQLpartitioning

Thepartition_maintenanceprocedure

Thepartition_createprocedure

Thepartition_verifyprocedure

Thepartition_dropprocedure

Thepartition_maintenance_allprocedure

Housekeepingconfiguration

B.CollectingSquidMetrics

Squidmetricscript

www.it-ebooks.info

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthors,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:February2015

Productionreference:1210215

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78439-976-4

www.packtpub.com

www.it-ebooks.info

CreditsAuthors

AndreaDalleVacche

StefanoKewanLee

Reviewers

RaviBhure

NicholasPier

NicolaVolpini

CommissioningEditor

AmarabhaBanerjee

AcquisitionEditor

NikhilKarkal

ContentDevelopmentEditor

SiddheshSalvi

TechnicalEditor

HumeraShaikh

CopyEditor

SarangChari

ProjectCoordinator

KrantiBerde

Proofreaders

SimranBhogal

LindaMorris

Indexer

HemanginiBari

Graphics

DishaHaria

ProductionCoordinator

AparnaBhagat

CoverWork

AparnaBhagat

www.it-ebooks.info

AbouttheAuthorsAndreaDalleVaccheisahighlyskilledITprofessionalwithover14yearsofexperienceintheITindustryandbanking.HegraduatedfromUniversitàdegliStudidiFerrarawithaninformationtechnologycertification.ThislaidthetechnologyfoundationthatAndreahasbuiltoneversince.Andreahasacquiredvariousindustry-respectedaccreditations,whichincludeCisco,Oracle,RHCE,ITIL,andofcourse,Zabbix.Throughouthiscareer,hehasworkedinmanylarge-scaleenvironments,ofteninrolesthathavebeenverycomplex,onaconsultantbasis.Thishasfurtherenhancedhisgrowingskillset,addingtohispracticalknowledgebaseandincreasinghisappetitefortheoreticaltechnicalstudying.

Andrea’sloveforZabbixcamefromhistimespentintheOracleworldasadatabaseadministrator/developer.Histimewasspentmainlyonreducingownershipcosts,specializinginmonitoringandautomation.ThisiswherehecameacrossZabbixandtheflexibilityitoffered,bothtechnicallyandadministratively.Withthisasalaunchpad,AndreawasinspiredtodevelopOrabbix,thefirstopensourcesoftwaretomonitorOracle’scompleteintegrationwithZabbix.HehaspublishedanumberofarticlesonZabbix-relatedsoftware,suchasDBforBIX.Hisprojectsarepubliclyavailableathttp://www.smartmarmot.com.Currently,Andreaisworkingasaseniorarchitectforaleadingglobalinvestmentbankinaverydiverseandchallengingenvironment.HedealswithmanyaspectsoftheUnix/Linuxplatformsaswellasmanytypesofthird-partysoftware,whicharestrategicallyalignedtothebank’stechnicalroadmap.Inadditiontothistitle,AndreaDalleVaccheisacoauthorofMasteringZabbix,PacktPublishing.

StefanoKewanLeeisanITconsultantwithmorethan12yearsofexperienceinsystemintegration,security,andadministration.HeisacertifiedZabbixspecialistinlargeenvironmentsholdsaLinuxadministrationcertificationfromtheLPIandaGIACGCFWcertificationfromSANSInstitute.Whenhe’snotbusybreakingwebsites,helivesinthecountrysidewithhistwocatsandtwodogsandpracticesmartialarts.Inadditiontothistitle,StefanoKewanLeeisacoauthorofMasteringZabbix,PacktPublishing.

www.it-ebooks.info

AbouttheReviewersRaviBhureisbasicallyanITengineerwithnicheskills,suchasChef,CloudAnsible,SaltStack,Python,Ruby,andShell/Bash.Healsowritescodeforinfrastructure,dailyIToperations,andsoon.Inshort,heisfondofusinghisskillsandknowledgeoffault-tolerantsolutionsfortheday-to-daymaintenanceofmission-criticalproductioninfrastructure.

Ravistartedinteractingwithcomputerssince1996whenhegothisfirstcomputerathome.Thingschangedveryfast,andin1998,heenteredthemagicalworldoftheInternet☺forthefirsttimeever,whichchangedhislife!Hestartedhisowncybercafein1999.In2004,hegothisfirstjobasafieldengineer,hiredtomaintainandsupportVRIUFOsystems.After2years,hemovedtoPuneandworkedwithmanyorganizations,suchasVyomLabs,GlamIndia,Symphony,andDhingana.

Themosthappeningandinterestingfactabouthisdiverseexposureisthatheisfromanartsbackground.Yes,heholdsabachelor’sdegreeinartsfromSRTMUniversity,Nanded,Maharashtra,India.Andweallwillhavetoagreethathehasthearttosolveproblems☺,agreatinspirationforpeoplewhoarenonengineers!

Currently,RaviisassociatedwithOpexSoftwareasaseniorDevOpsengineer.

NicholasPierisanetworkengineerinthemanagedservices/professionalservicesfield.HisexperienceincludesdesigningdatacenternetworkinfrastructureswithvirtualizationandSANsolutions,webdevelopment,andwritingmiddlewareforbusinessapplications.Atthetimeofwritingthis,Nicholasholdsanumberofindustrycertifications,includingtheCiscoCCNP,VMwareVCP5-DCV,andvariousotherCiscoandCompTIAcertifications.Inhisfreetime,heindulgesinhispassionforcraftbeer,distancerunning,andreading.

I’dliketothankPacktPublishingforthisopportunity!

NicolaVolpinihasbeenplayingwithtechnologyfromayoungage,havingahardtimeresistingtheurgetodisassemblecomplextoysorkitchenappliances.

Theloveforcomputersoriginatedaroundhistenthbirthday,whenheaccidentallytoastedhisfirstCPU.Thisepisodeonlyincreasedhisfascinationforcomputers,andtheaccidents,fortunately,stopped.

Forthepast10years,he’sbeenworkingasanITprofessional,specializinginenterprisenetworkingandsystemadministration.ExperimentingwiththemostdiversetechnologiesinthefieldandbeinganavidfanoftheFOSSphilosophy,Linux,and*BSD,hedreamsofseeingthecollaborativethinkingoftheFOSSmovementhelpinspiretheworld.

He’scurrentlyworkingatStockholm,Sweden,whereheresideswithhisgirlfriend.

www.it-ebooks.info

www.PacktPub.com

www.it-ebooks.info

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

www.it-ebooks.info

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

www.it-ebooks.info

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

www.it-ebooks.info

PrefaceNetworkadministratorsarefacinganinterestingchallengethesedays.Ontheonehand,computernetworksarenotsomethingnewanymore.Theyhavebeenaroundforquiteawhile:theirphysicalcomponentsandcommunicationprotocolsarefairlywellunderstoodanddon’trepresentabigmysterytoanincreasingnumberofprofessionals.Moreover,networkappliancesaregettingcheaperandeasiertosetup,tothepointthatitdoesn’ttakeacertifiedspecialisttoinstallandconfigureasimplenetworkorconnectittoothernetworks.Theveryconceptofnetworkingissowidespreadandingrainedinhowusersanddevelopersthinkofacomputersystemthatbeingonlineinsomeformisexpectedandtakenforgranted.Inotherwords,acomputernetworkisincreasinglyseenasacommodity.

Ontheotherhand,theverysameforcesthatarecallingforsimpler,easier,accessiblenetworksaretheonesthatareactuallypushingthemtogrowmoreandmorecomplexeveryday.It’samatterofbothquantityandquality.Thenumberofconnecteddevicesonagivennetworkisalmostalwaysconstantlygrowingandsoistheamountofdataexchanged:mediastreams,applicationdata,backups,databasequeries,andreplicationtendtosaturatebandwidthjustasmuchastheyeatupstoragespace.Asforquality,therearedozensofdifferentrequirementsthatfactorinagivennetworksetup:fromhavingtomanagedifferentphysicalmediums(fiber,cable,radio,andsoon),totheneedtoprovidehighperformanceandavailability,bothontheconnectionandontheapplicationlevel;fromtheneedtoincreaseperformanceandreliabilityforgeographicallinks,toprovidingconfidentiality,security,anddataintegrityatalllevels,andthelistgoeson.

Thesetwocontrasting,yetintertwined,tendenciesareforcingnetworkadministratorstodomore(moreservices,moreavailability,andmoreperformance)withless(lessbudget,butalsolessattentionfromthemanagementcomparedtonewer,flashiertechnologies).Now,morethanever,asanetworkadmin,youneedtobeabletokeepaneyeonyournetworkinordertokeepitinahealthystate,butalsotoquicklyidentifyandresolvebottlenecksandoutagesofanykind—orbetteryet,findwaystoanticipateandworkaroundthembeforetheyhappen.You’llalsoneedtointegrateyoursystemswithdifferenttoolsandenvironments(bothlegacyandstrategicones)thatwillbeoutofyourdirectcontrol,suchasassetdatabases,incidentmanagementsystems,accountingandprofilingsystems,andsoon.Evenmoreimportantly,you’llneedtobeabletoshowyourworkandexplainyourneedsinclear,understandabletermstonontechnicalpeople.

Now,ifweweretosaythatZabbixistheperfect,one-size-fits-allsolutiontoallyournetworkmonitoringandmanagementproblems,wewouldclearlybelying.Tothisday,nosuchtoolexistsdespitewhatmanyvendorswantyoutobelieve.Eveniftheyhavemanyfeaturesincommon,whenitcomestomonitoringandcapacitymanagement,everynetworkhasitsownquirks,specialcases,andpeculiarneeds,tothepointthatanytoolhastobecarefullytunedtotheenvironmentorfacetheriskofbecominguselessandneglectedveryquickly.

WhatistrueisthatZabbixisamonitoringsystempowerfulenoughandflexibleenough

www.it-ebooks.info

that,withtherightamountofwork,canbecustomizedtomeetyourspecificneeds.Andagain,thoseneedsarenotlimitedtomonitoringandalerting,butalsotoperformanceanalysisandprediction,SLAreporting,andsoon.WhenusingZabbixtomonitoranenvironment,youcancertainlycreateitemsthatrepresentvitalmetricsforthenetworkinordertohaveareal-timepictureofwhat’shappening.However,thosesameitemscanalsoproveveryusefultoanalyzeperformancebottlenecksandtoplannetworkexpansionandevolution.Items,triggers,andactionscanworktogethertoletyoutakeanactiveroleinmonitoringyournetworkandeasilyidentifyandpre-emptcriticaloutages.

Inthisbook,we’llassumethatyoualreadyknowZabbixasageneral-purposemonitoringtool,andthatyoualsousedittoacertainextent.Specifically,wewon’tcovertopicssuchasitem,trigger,oractioncreationandconfigurationwithabasic,step-by-stepapproach.Here,wewanttofocusonafewtopicsthatcouldbeofparticularinterestfornetworkadministrators,andwe’lltrytohelpthemfindtheirownanswerstoreal-worldquestionssuchasthefollowing:

Ihavealargenumberofappliancestomonitorandhavetokeepmonitoringdataavailableforalongtimeduetoregulatoryrequirements.HowdoIinstallandconfigureZabbixsothatitisabletomanageeffectivelythislargeamountofdata?Whatarethebestmetricstocollectinordertobothhaveaneffectivereal-timemonitoringsolutionandleveragehistoricaldatatomakeperformanceanalysisandpredictions?ManyZabbixguidesandtutorialsfocusonusingtheZabbixagent.Theagentiscertainlypowerfulanduseful,buthowdoIleverageinaneffectiveandsecurewaymonitoringprotocolsthatarealreadyavailableonmynetwork,suchasSNMPandnetflow?Loadbalancers,proxies,andwebserverssometimesfallunderagrayareabetweennetworkandapplicationadministration.Ihaveabunchofwebserversandproxiestomonitor.Whatkindofmetricsaremostusefultocheck?Ihaveacomplexnetworkwithhoststhataredeployedanddecommissionedonadailybasis.HowdoIkeepmymonitoringsolutionup-to-datewithoutresortingtolong,error-pronemanualinterventionsasmuchaspossible?NowthatIhavecollectedalargeamountofmonitoringandperformancedata,howcanIanalyzeitandshowtheresultsinameaningfulway?HowdoIputtogetherthegraphsIhaveavailabletoshowhowtheyarerelated?

Inthecourseofthenextfewchapters,we’lltrytoprovidesomepointersonhowtoanswerthosequestions.Wediscussasmanypracticalexamplesandreal-worldapplicationsaswecanaroundthesubjectofnetworkmonitoring,butmorethananything,wewantedtoshowyouhowit’srelativelysimpletoleverageZabbix’spowerandflexibilitytoyourownneeds.

Theaimofthisbookisnottoprovideyouwithasetofprepackagedrecipesandsolutionsthatyoucanapplyuncriticallytoyourownenvironment.Eventhoughweprovidedsomescriptsandcodethataretestedandworking(andhopefullyyou’llfindthemuseful),therealintentionwasalwaystogiveyouadeeperunderstandingofthewayZabbixworksso

www.it-ebooks.info

thatyouareabletocreateyourownsolutionstoyourownchallenges.

Wehopewehavesucceededinourgoal,andthatbytheendofthebook,you’llfindyourselfamoreconfidentnetworkadministratorandamoreproficientZabbixuser.Evenifthiswillnotbethecase,wehopeyou’llbeabletofindsomethingusefulinthefollowingchapters:wetouchupondifferentaspectsofZabbixandnetworkmonitoringandalsodiscussacoupleoflessknownfeaturesthatyoumightfindveryinterestingnonetheless.

So,withoutfurtherado,let’sgetstartedwiththeactualcontentwewanttoshowyou.

www.it-ebooks.info

WhatthisbookcoversChapter1,InstallingaDistributedZabbixSetup,teachesyouhowtoinstallZabbixinadistributedsetup,withalargeuseofproxies.Thechapterwillguideyouthroughallthepossiblesetupscenarios,showingyouthemaindifferencesbetweentheactiveandpassiveproxysetup.ThischapterwillexplainhowtoprepareandsetupaZabbixinstallation,whichisreadytobegrownwithinyourinfrastructure,readytosupportyou,andmonitoralargeenvironmentorevenaverylargeone.

Chapter2,ActiveMonitoringofYourDevices,offersyouafewveryusefulexamplesofthedifferentmonitoringpossibilitiesZabbixcanachievebyrelyingondifferentmethodsandprotocols.You’llseehowtoqueryyournetworkfromthelinkleveluptoroutingandnetworkflowusingICMP,SNMP,andlog-parsingfacilitiestocollectyourmeasurements.Youwillalsolearnhowtoextractmeaningfulinformationfromthegathereddatausingaggregatedandcalculateditems,andconfiguringcomplextriggersthatwillalertyouaboutrealnetworkissueswhileminimizingsignalnoiseandfalsepositives.

Chapter3,MonitoringYourNetworkServices,takesyouthroughhowtoeffectivelymonitorthemostcriticalnetworkservices,suchasDNS,DHCP,NTP,Apacheproxy/reverseproxies,andproxycacheSquid.Asitiseasytounderstand,allofthemarecriticalserviceswhereasimpleissuecanaffectyournetworksetupandquicklypropagatetheissuetoyourentirenetwork.Youwillunderstandhowtoextractmeaningfulmetricsandusefuldatafromallthelistedservices,beingablethennotonlytomonitortheirownreliability,butalsotoacquireimportantmetricsthatcanhelpyoutopredictfailuresorissues.

Chapter4,DiscoveringYourNetwork,explainshowtodeeplyautomatethemonitoringconfigurationofnetworkobjects.Itwillmassivelyusethebuilt-indiscoveryfeatureinordertokeepthemonitoringsolutionup-to-datewithinanevolvingnetworkenvironment.ThischapterisdividedintotwocorepartsthatcoverthetwomainlevelsofZabbix’sdiscovery:hostdiscoveryandlow-leveldiscovery.

Chapter5,VisualizingYourTopologywithMapsandGraphs,showsyouhowtocreatecomplexgraphsfromyouritem’snumericalvalues,automaticallydrawmapsthatreflectthecurrentstatusofyournetwork,andbringitalltogetherusingscreensasatooltocustomizemonitoringdatapresentation.ThischapteralsopresentsasmartwaytoautomatetheinitialstartupofyourZabbix’ssetup,makingyouabletodrawnetworkdiagramsusingmapsinafullyautomatedway.Youwillthenlearnaproduction-readymethodtomaintainmapswhileyournetworkisgrowingorrapidlychanging.

AppendixA,PartitioningtheZabbixDatabase,containsalltherequiredsoftwareandstoredprocedurestoefficientlypartitionyourZabbixdatabase.

AppendixB,CollectingSquidMetrics,containsthesoftwareusedtomonitorSquid.

www.it-ebooks.info

WhatyouneedforthisbookThesoftwarethathasbeenusedandisnecessaryforthisbookis:

LinuxRedHatEnterpriseLinux6.5orhigherZabbix4.2ApacheHTTPD2.2MySQLServer-5.1Netflow1.6.12Nmap

Thisbookalsorequiresanintermediateexperienceinshellscripting,abasic-to-intermediateknowledgeofPython,andanintermediateknowledgeofZabbix.

Anyway,alltheexamplesdiscussedandproposedinthisbookareexplainedwellandcommentedupon.Thesameapproachhasbeenappliedeventothesoftwareusedonthisbookwhereitisexplained,withareasonablelevelofdetail,howtosetupandconfigureeachsoftwarecomponent.

www.it-ebooks.info

WhothisbookisforThisbookisintendedforexperiencednetworkadministratorslookingforacomprehensivemonitoringsolutionfortheirnetworks.ThereadermusthaveagoodknowledgeofUnix/Linux,networkingconcepts,protocols,andappliancesandabasic-to-intermediateknowledgeofZabbix.Thereaderwillbeguidedstepbysteptomanageandleadalltheimportantpointsyouwillhavetodealwith.Youwillthenbeabletostartupaneffectiveandlarge-environment-readyZabbixmonitoringsolutionthatwillbeaperfectfitwithinyournetwork.

www.it-ebooks.info

ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“OntheZabbixserver-side,youneedtocarefullysetthevalueofStartTrappers=.”

Ablockofcodeissetasfollows:

#FirstofallweneedtoimportcsvandNetworkx

importcsv

importnetworkxasnx

#Thenweneedtodefinewhoisourzabbixserverandsomeotherdetailto

properlyproducetheDOTfile

zabbix_service_ipaddr="192.168.1.100"

main_loop_ipaddr="10.12.20.1"

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

#wecanopenourCSVfile

csv_reader=csv.DictReader(open('my_export.csv'),\

delimiter=",",\

fieldnames=("ipaddress","hostname","oid","dontcare","neighbors"))

#Skiptheheader

csv_reader.next()

Anycommand-lineinputoroutputiswrittenasfollows:

#chkconfig--level345zabbix-serveron

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“Thereisaclearwarningonthewebsitethatwarnsuswiththisstatement:TheApplianceisnotintendedforseriousproductionuseatthistime.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

www.it-ebooks.info

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<feedback@packtpub.com>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.

www.it-ebooks.info

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

www.it-ebooks.info

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

www.it-ebooks.info

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

www.it-ebooks.info

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.

www.it-ebooks.info

QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<questions@packtpub.com>,andwewilldoourbesttoaddresstheproblem.

www.it-ebooks.info

Chapter1.InstallingaDistributedZabbixSetupMostlikely,ifyouarereadingthisbook,youhavealreadyusedandinstalledZabbixasanetworkmonitoringsolution.Now,inthischapter,wewillseehowtoinstallZabbixinadistributedsetup,eventuallymovingontoalargeuseofproxies.Thechapterwilltakeyouthroughallthepossiblescenariosandexplainthemaindifferencesbetweentheactiveandpassiveproxysetup.Usually,thefirstZabbixinstallationisdoneasapartoftheconcepttoseewhethertheplatformisgoodenoughforyou.Here,thecommonerroristostartusingthissetuponalargeproductionenvironment.Afterreadingthischapter,youwillbereadytoinstallandsetupalargeenvironmentreadyinfrastructure.

Inthischapter,wewillexplainhowtoprepareandsetupaZabbixinstallation,whichisreadytobegrownwithinyourinfrastructure,andreadyforalargetoaverylargeenvironment.ThisbookismainlyfocusedonZabbixfornetworkmonitoring.Thischapterwillquicklytakeyouthroughtheinstallationprocess,emphasizingonallthemostimportantpointsyouneedtoconsider.Inthenextchapter,wewillspendmoretimedescribingabetterapproachtomonitoryournetworkdevicesandhowtoretrieveallthecriticalmetricsfromthem.Afterreadingthischapter,youwillbecomeawareofthecommunicationbetweenserverandproxiesbeingabletomixtheactiveandpassivesetupinordertoimproveyourinfrastructure.YoucanextendthestrongcentralZabbixcoresetupwithmanylightweightandeffectiveZabbixproxiesactingasasatelliteinsideyournetworktoimproveyourmonitoringsystem.

www.it-ebooks.info

ZabbixarchitecturesZabbixwasbornasadistributednetworkmonitoringtoolwithacentralwebinterfacewhereyoucanmanagealmosteverything.Nowadays,withZabbix2.4,thenumberofpossiblearchitectureshasbeenreducedtoasingleserversetupandaZabbix-proxiesdistributedsetup.

NoteFromZabbix2.4,thenode-setupwasdiscontinued.Moreinformationisavailableathttps://www.zabbix.com/documentation/2.4/manual/introduction/whatsnew240#node-based_distributed_monitoring_removed.

Now,thesimplestarchitecture(whichisreadytohandlelargeenvironmentssuccessfully)thatyoucanimplementcomposesofthreeservers:

WebserverRDBMSserverZabbixserver

Topreparethissimplesetupforalargeenvironmentsetting,it’sbettertouseadedicatedserverforeachoneofthesecomponents.

Thisisthesimplestsetupthatcanbeeasilyextendedandisreadytosupportalargeenvironment.

Theproposedarchitectureisshowninthefollowingdiagram:

ThiskindofsetupcanbeextendedbyaddingmanyZabbixproxiesresultinginaproxy-basedsetup.Theproxy-basedsetupisimplementedwithoneZabbixserverandseveralproxies:oneproxyperbranch,datacenteror,inourcase,foreachremotenetworksegmentyouneedtomonitor.

Thisconfigurationiseasytomaintainandofferstheadvantagetohaveacentralizedmonitoringsolution.Thiskindofconfigurationistherightbalancebetweenlargeenvironmentmonitoringandcomplexity.

www.it-ebooks.info

TheZabbixproxy,likeaserver,isusedtocollectdatafromanynumberofhostsordevices,acquiringallthemetricsrequestedandactingasaproxy.Thismeansthatitcanretainthisdataforanarbitraryperiodoftime,relyingonadedicateddatabasetodoso.Theproxydoesn’thaveafrontendandismanageddirectlyfromthecentralserver.

NoteTheproxylimitsitselftodatacollectionwithouttriggerevaluationsoractions;allthedataisstoredinitsdatabase.Forthisreason,it’sbettertouseanefficientrobustRDBMSthatcanpreventdatalossincaseofacrash.

AllthesecharacteristicsmaketheZabbixproxyalightweighttooltodeployandoffloadsomechecksfromthecentralserver.Ourobjectiveistocontrolandstreamlinetheflowofmonitoreddataacrossnetworks,andtheZabbixproxygivesusthepossibilitytosplitandsegregateitemsanddataonthedifferentnetworks.Themostimportantfeatureisthattheacquiredmetricsarestoredinitsdatabase.Therefore,incaseofanetworkloss,youwillnotlosethem.

www.it-ebooks.info

UnderstandingZabbixdataflowThestandardZabbixdataflowiscomposedofseveralactorsthatsenddatatoourZabbixserver.OfallthesourcesthatcansenddatatoourZabbixserver,wecanidentifythreemaindatasources:

ZabbixagentZabbixsenderOtheragents(externalscriptsorcomponentsbuiltinhouse)

Theotheragentsrepresentedinthenextdiagramcanbeoftwomaintypes:

Customand/orthird-partyagentsZabbixproxy

Asthediagramdisplaysthedatathatgetsacquiredfrommanydifferentsourcesintheformofitems.Attheendofthediagram,youseetheGUI,whichpracticallyrepresentstheusersconnectedandthedatabasethatistheplacewhereallthevaluesarestored.

Inthenextsection,wewilldivedeepintotheZabbixproxies’dataflow.

www.it-ebooks.info

UnderstandingtheZabbixproxies’dataflowZabbixproxiescanoperateintwodifferentmodes,activeandpassive.Thedefaultsetupistheactiveproxy.Inthissetup,theproxyinitiatesallconnectionstotheZabbixserver,theoneusedtoretrieveconfigurationinformationonmonitoredobjects,andtheconnectiontosendmeasurementsbacktotheserver.Here,youcanchangeandtweakthefrequencyofthesetwoactivitiesbysettingthefollowingvariablesintheproxyconfigurationfile:/etc/zabbix/zabbix_proxy.conf:

ConfigFrequency=3600

DataSenderFrequency=1

Valuesareexpressedinseconds.OntheZabbixserver-side,youneedtocarefullysetthevalueofStartTrappers=.

Thisvalueneedstobegreaterthanthenumberofallactiveproxiesandnodesyoudeployed.Thetrapperprocesses,indeed,managealltheincominginformationfromtheproxies.

NotePleasenotethattheserverwillforkextraprocessesasrequired,ifneeded,butitisstronglyadvisabletopreforkalltheprocessesthatareneededduringthestartup.Thiswillreducetheoverheadduringthenormaloperation.

Ontheproxyside,anotherparametertoconsideris:

HeartbeatFrequency

Thisparametersetsasortofkeepalive,whichafterthedefinednumberofseconds,willcontacttheserveralthoughitdoesn’thaveanydatatosend.Theproxyavailabilitycanbeeasilycheckedwiththefollowingitem:

zabbix[proxy,"proxyuniquename",lastaccess]

Heretheproxyuniquename,ofcourse,istheidentifieryouassignedtotheproxyduringdeployment.Theitemwillreturnthenumberofsecondsasthelasttimethattheproxywascontacted,avalueyoucanthenusewiththeappropriatetriggeringfunctions.

TipIt’sreallyimportanttohaveatriggerassociatedtothisitem,soyoucanbewarnedincaseofconnectionloss.Lookingatthetrendofthistrigger,youcanlearnaboutaneventualreapingtimesetonthefirewall.Let’slookatapracticalexample:ifyounoticethatafter5minutesyourconnectionsaredropped,settheheartbeatfrequencyto120secondsandcheckforthelastaccesstimeabove300seconds.

Inthefollowingdiagram,youcanseethecommunicationflowbetweentheZabbixserverandtheproxy:

www.it-ebooks.info

Asyoucanseefromthediagram,theserverwillwaittoreceiverequestsfromtheproxyandnothingmore.

NoteTheactiveproxyisthemostefficientwaytooffloaddutiesfromtheserver.Indeed,theserverwilljustsitherewaitingtobeaskedaboutchangesinconfiguration,ortoreceivenewmonitoringdata.

Ontheotherside,proxiesareusuallydeployedtomonitorsecurenetworksegmentswithstrictoutgoingtrafficpolicies,andareusuallyinstalledonDMZs.Inthesekindofscenarios,normally,itisverydifficulttoobtainpermissionfortheproxytoinitiatethecommunicationwiththeserver.Unfortunately,it’snotjustduetopolicies.DMZsareisolatedasmuchaspossiblefrominternalnetworks,astheyneedtobeassecureastheycan.Generally,it’softeneasierandmoreacceptedfromasecuritypointofviewtoinitiateaconnectionfromtheinternalnetworktoaDMZ.Inthiskindofscenario,thepassiveproxyisveryhelpful.Thepassiveproxyisalmostamirroredimageoftheactiveproxysetup,asyoucanseeinthefollowingdiagram:

Withthisconfiguration,theZabbixserverwillcontacttheproxyperiodicallytodelivertheconfigurationchangesandtorequesttheitemvaluestheproxyisholding.

Thisistheproxyconfigurationtoenabletheproxyyouneedtoset:

www.it-ebooks.info

ProxyMode=1

Thisparameterspecifiesthepassiveproxy,youdon’tneedtodoanythingelse.Now,ontheserverside,youneedtosetthefollowingparameters:

StartProxyPollers=

Thiswillsetthenumberofprocessesdedicatedtothepassiveproxies

NoteTheStartProxyPollersparametershouldmatchthenumberofpassiveproxiesyouhavedeployed.

ProxyConfigFrequency=

Thisvalueexpressesthefrequencywithwhichtheserversendstheconfigurationtoitsproxy

ProxyDataFrequency=

Thisistheintervalparameterthatexpressesthenumberofsecondsbetweentwoconsecutiverequeststogettheacquiredmetricsfromtheproxy

Theitemusedtocheckapassiveproxy’savailabilityisasfollows:

zabbix[proxy,"proxyuniquename",lastaccess]

Thisisexactlythesameastheactiveone.

Thepassiveproxyenablesustogathermonitoringdatafromotherwiseclosedandlockeddownnetworkswithaslightlyincreasedoverhead.

NoteYoucanmixasmanyactiveandpassiveproxiesasyouwantinyourenvironment.Thisenablesyoutoexpandyourmonitoringsolutiontoreacheachpartofthenetworkandtohandlealargenumberofmonitoredobjects.Thisapproachkeepsthearchitecturesimpleandeasytomanagewithastrongcentralcoreandmanysimple,lightweightsatellites.

Ifyouwouldliketokeeptrackofalltheremainingitemsthattheproxyneedstosend,youcansetuptheproxytorunthisqueryagainstitsdatabase:

SELECT((SELECTMAX(proxy_history.id)FROMproxy_history)-nextid)FROMids

WHEREfield_name='history_lastid'

TipDownloadingtheexamplecode

Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

ThisquerywillreturnthenumberofitemsthattheproxystillneedstosendtotheZabbix

www.it-ebooks.info

server.ConsideringthatyouareusingMySQLasadatabase,youneedtoaddthefollowinguserparameterintheproxyagentconfigurationfile:

UserParameter=zabbix.proxy.items.sync.remaining,mysql-u<yourdbname

here>-p'<yourpasswordhere>'-e'SELECT((SELECTMAX(proxy_history.id)

FROMproxy_history)-nextid)FROMidsWHEREfield_name=history_lastid'2>&1

Now,allyouneedtodoissetanitemontheZabbixserversideandyoucanseehowyourproxyisfreeingitsqueue.

www.it-ebooks.info

InstallingZabbixZabbix,likealltheothersoftware,canbeinstalledintwoways:

1. Downloadthelatestsourcecodeandcompileit.2. Installitfrompackages.

Actually,thereisanotherwaytohaveaZabbixserverupandrunning:usingthevirtualappliance.TheZabbixserverappliancewillnotbeconsideredinthisbookasZabbixitselfdefinesthisvirtualapplianceasnotreadyforproductiveenvironments.Thisvirtualapplianceisnotaproductionreadysetupformanyreasons:

Itisamonolithwhereeverythingisinstalledonthesameserver.Thereisnoseparationfromthedatabaselayerandthepresentationlayer.Thismeansthateachoneofthesecomponentscanaffecttheperformanceoftheother.Thereisaclearwarningonthewebsitethatwarnsuswiththisstatement:TheApplianceisnotintendedforseriousproductionuseatthistime.

Ontheotherhand,theinstallationfrompackagesgivesussomebenefits:

ThepackagesmakeiteasytoupgradeandupdateDependenciesareautomaticallysortedout

Thesourcecodecompilationalsogivesussomebenefits:

WecancompileonlytheneededfeaturesWecanbuildtheagentstaticallyanddeployondifferentLinuxflavorsCompletecontrolonupdate

It’squiteusualtohavedifferentversionsofLinux,Unix,andMicrosoftWindowsonalargeenvironment.Thiskindofscenarioisquitediffusedonaheterogeneousinfrastructure,andifweusetheZabbix’sagentdistributionpackageoneachLinuxserver,wewillhavedifferentversionsoftheagentforsure,anddifferentlocationsfortheconfigurationfiles.

Themorethethingsarestandardizedacrossourserver,theeasieritwillbecometomaintainandupgradetheinfrastructure.The--enable-staticoptiongivesusawaytostandardizetheagentacrossdifferentLinuxversionsandrelease,whichisastrongbenefit.Theagent,staticallycompiled,canbeeasilydeployedeverywhereand,forsure,wewillhavethesamelocation(andwecanusethesameconfigurationfileapartfromthenodename)fortheagentandhis/herconfigurationfile.Theonlythingthatmightvaryisthestart/stopscriptandhowtoregisteritontherightinitrunlevel,butatleastthedeploymentwillbestandardized.

ThesamekindofconceptcanbeappliedtothecommercialUnix,bearinginmindtocompileitonthetargetenvironmentsothatthesameagentcanbedeployedondifferentUnixreleasesofthesamevendor.

www.it-ebooks.info

InstallingfrompackagesThefirstthingtodotoinstallZabbixfromrepoistoaddtheyumrepositorytoourlist.Thiscanbedonewiththefollowingcommand:

$rpm-Uvhhttp://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-

2.4-1.el6.noarch.rpm

Retrievinghttp://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-

2.4-1.el6.noarch.rpm

warning:/var/tmp/rpm-tmp.dsDB6k:HeaderV4DSA/SHA1Signature,keyID

79ea5ed4:NOKEY

Preparing…###########################################[100%]

1:zabbix-release###########################################[100%]

Oncethisisdone,wecantakeadvantageofallthebenefitsintroducedbythepackagemanagerandhavethedependenciesautomaticallyresolvedbyyum.

ToinstalltheZabbixserver,yousimplyneedtorun:

$yuminstallzabbix-server-mysqlzabbix-agentzabbix-javagateway

Now,youhaveyourserverreadytostart.Wecan’tstartitnowasweneedtosetupthedatabase,whichwillbedoneinthenextheading,anyway,whatyoucandoissetupthestart/stoprunlevelforourzabbix_serverandzabbix_agentdaemons:

$chkconfig--level345zabbix-serveron

$chkconfig--level345zabbix-agenton

Pleasedoublecheckifthepreviouscommandransuccessfullywiththefollowing:

$chkconfig--list|grepzabbix

zabbix-agent0:off1:off2:off3:on4:on5:on6:off

zabbix-server0:off1:off2:off3:on4:on5:on6:off

www.it-ebooks.info

SettingupaZabbixagentNow,asusuallyhappensinalargeserverfarm,itispossiblethatyouhavemanydifferentvariantsofLinux.Here,ifyoucan’tfindthepackageforyourdistribution,youcaneventhinktocompiletheagentfromscratch.Thefollowingarethestepsforthesame:

1. DownloadthesourcecodefromtheZabbixwebsite.2. Unpackthesoftware.3. Satisfyallthesoftwaredependencies,installingalltherelated-develpackages.4. Runthefollowingcommand:$./configure--enable-agent.

TipHere,youcanstaticallylinktheproducedbinarywiththe--enable-staticoption.Withthis,thebinaryproducedwillnotrequireanyexternallibrary.ThisisreallyusefultodistributetheagentacrossdifferentversionsofLinux.

Compileeverythingwith$make.

Now,beforeyourun$makeinstall,youcandecidetocreateyourownpackagetodistributewithCheckInstall.

www.it-ebooks.info

CreatingaZabbixagentpackagewithCheckInstallTheadviceistonotrunmakeinstall,butuseCheckInstalltoproducetherequiredpackageforyourLinuxOSfromhttp://asic-linux.com.mx/~izto/checkinstall/.

NoteWecanalsouseaprebuiltCheckInstall;thecurrentreleaseischeckinstall-1.6.2-20.2.i686.rpmonRedHat/CentOS.Thepackagewillalsoneedtherpm-buildpackage:

rpm-buildyuminstall

Also,weneedtocreatethenecessarydirectories:

mkdir-p~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

Thissoftwareenablesyoutocreateapackageformanydifferentversionsofthepackagemanager,namely,RPM,deb,andtgz.

NoteCheckInstallwillproducepackagesforDebian,Slackware,andRedHat,helpingustopreparetheZabbix’sagentpackage(staticallylinked)anddistributeitaroundourserver.

Now,weneedtoswitchtotherootaccountusing$sudosu–.Also,usethecheckinstallfollowedbytheseoptions:

$checkinstall--nodoc--install=yes-y

Ifyoudon’tfaceanyissue,youshouldgetthefollowingmessage:

******************************************************************

Done.Thenewpackagehasbeensavedto

/root/rpmbuild/RPMS/i386/zabbix-2.4.0-1.i386.rpm

Youcaninstallitinyoursystemanytimeusing:

rpm-izabbix-2*.4.0-1.i386.rpm

******************************************************************

Rememberthattheserverbinarieswillbeinstalledin<prefix>/sbin,utilitieswillbein<prefix>/bin,andthemainpagesunderthe<prefix>/sharelocation.

TipTospecifyadifferentlocationforZabbixbinaries,weneedtouse--prefixontheconfigureoptions(forexample,--prefix=/opt/zabbix).

www.it-ebooks.info

ServerconfigurationFortheserverconfiguration,weonlyhaveonefiletocheckandedit:

/etc/zabbix/zabbix_server.conf

Alltheconfigurationfilesarecontainedinthefollowingdirectory:

/etc/zabbix/

Allyouneedtochangefortheinitialsetupisthe/etc/zabbix/zabbix_server.confconfigurationfileandwritetheusername/passwordanddatabasenamehere.

NotePleasetakecaretoprotecttheaccesstotheconfigurationfilewithchmod400/etc/zabbix/zabbix_server.conf.

Thedefaultexternalscriptslocationis:

/usr/lib/zabbix/externalscripts

Also,thealertscriptdirectoryis:

/usr/lib/zabbix/alertscripts

Thiscanbechangedbyeditingthezabbix_server.conffile.

Theconfigurationontheagentsideisquiteeasy;basically,weneedtowritetheIPaddressofourZabbixserver.

www.it-ebooks.info

InstallingadatabaseThedatabasewewilluseonthisbook,asalreadyexplained,isMySQL.

Now,consideringthatyouhaveaRedHatserver,theproceduretoinstallMySQLfromtheRPMrepositoryisquiteeasy:

$yuminstallmysqlmysql-server

Now,youneedtosetuptheMySQLservicetostartautomaticallywhenthesystemboots:

$chkconfig--levels235mysqldon

$/etc/init.d/mysqldstart

TipRemembertosetapasswordfortheMySQLrootuser

Tosetapasswordfortheroot,youcanrunthesetwocommands:

/usr/bin/mysqladmin-urootpassword'new-password'

/usr/bin/mysqladmin-uroot-hhostname-of-your.zabbix.dbpassword'new-

password'

Alternatively,youcanrun:

/usr/bin/mysql_secure_installation

Thiswillalsohelpyoutoremovethetestdatabasesandanonymoususerdatathatwascreatedbydefault.Thisisstronglyrecommendedforproductionservers.

Now,it’stimetocreatetheZabbixdatabase.Forthis,wecanusethefollowingcommands:

$mysql-uroot-p

$mysql>CREATEDATABASEzabbixCHARACTERSETUTF8;

QueryOK,1rowaffected(0.00sec)

$mysql>GRANTALLPRIVILEGESonzabbix.*to'zabbixuser'@'localhost'

IDENTIFIEDBY'zabbixpassword';

QueryOK,0rowsaffected(0.00sec)

$mysql>FLUSHPRIVILEGES;

$mysql>quit

Next,weneedtorestorethedefaultZabbixMySQLdatabasefiles:

$mysql-uzabbixuser-pzabbixpasswordzabbix</usr/share/doc/zabbix-

server-mysql-2.4.0/create/schema.sql

server-mysql-2.4.0/create/images.sql

server-mysql-2.4.0/create/data.sql

Now,ourdatabaseisready.Beforewebegintoplaywiththedatabase,it’simportanttodosomeconsiderationaboutdatabasesizeandheavytasksagainstit.

www.it-ebooks.info

ConsideringthedatabasesizeZabbixusestwomaingroupsoftablestostoreitsdata:

HistoryTrends

Now,thespaceconsumedbythesetablesisinfluencedby:

Items:Thisisthenumberofitemsyou’regoingtoacquireRefreshrate:ThisisthemeanaveragerefreshrateofouritemsSpacetostorevalues:ThisdependsonRDBMS

Thespaceusedtostoredatacanvaryduetothedatabase,butwecanresumethespaceusedbythesetablesinthefollowingtable:

Typeofmeasure Retentionindays Spacerequired

History 30 10.8G

Events 1825(5years) 15.7GB

Trends 1825(5years) 26.7GB

Total NA 53.2GB

Thiscalculationis,ofcourse,doneconsideringtheenvironmentafter5yearsofretention.Anyway,weneedtohaveanenvironmentreadytosurvivethisperiodoftimeandretainthesameshapethatithadwhenitwasinstalled.Wecaneasilychangethehistoryandtrendsretentionpolicyperitem.Thismeansthatwecancreateatemplatewithitemsthathaveadifferenthistoryretentionbydefault.Normally,thehistoryissetto30days,butforsomekindofmeasure(suchasinwebscenarios)orotherparticularmeasures,weneedtokeepallthevaluesformorethanaweek.Thispermitsustochangethisvalueoneachitem.

www.it-ebooks.info

MySQLpartitioningNowthatweareawareofhowbigourdatabasewillbe,it’seasytoimaginethathousekeepingwillbeaheavytaskandthetime,CPU,andresourceconsumedbythisonewillgrowtogetherwiththedatabasesize.

Housekeepingisinchargetoremovetheoutdatedmetricsfromthedatabaseandtheinformationdeletedbyauser,andaswe’veseenthehistory,trends,andeventstablesare,aftersometime,hugetables.Thisexplainswhytheprocessissoheavytomanage.

Theonlywaywecanimproveperformancesoncewehavereachedthisvolumeofdataisbyusingpartitioninganddisablingthehousekeeperaltogether.

Partitioningthehistoryandtrendtableswillprovideuswithmanymajorbenefits:

Allhistorydatainatableforaparticulardefinedwindowtimeareself-containedinitsownpartition.Thisallowsyoutoeasilydeleteolddatawithoutimpactingthedatabaseperformance.WhenyouuseMySQLwithInnoDB,andifyoudeletedatacontainedinatable,thespaceisnotreleased.Thespacefreedismarkedasfree,butthediskspaceconsumedwillnotchange.Whenyouusepartition,andifyoudropapartition,thespaceisimmediatelyfreed.Queryperformancecanbeimproveddramaticallyinsomesituations,inparticular,whenthereisheavyaccesstothetable’srowsinasinglepartition.Whenaqueryupdatesahugeamountofdataorneedsaccesstoalargepercentageofthepartition,thesequentialscanisoftenmoreefficientthantheindexusagewitharandomaccessorscatteredreadsagainstthisindex.

Unfortunately,Zabbixisnotabletomanagethepartitions.So,weneedtodisablehousekeeping,anduseanexternalprocesstoaccomplishhousekeeping.

Whatweneedtohaveisastoredprocedurethatdoesalltheworkforus.

Thefollowingisthestoredprocedure:

DELIMITER$$

CREATEPROCEDURE`partition_maintenance`(SCHEMA_NAMEVARCHAR(32),

TABLE_NAMEVARCHAR(32),KEEP_DATA_DAYSINT,HOURLY_INTERVALINT,

CREATE_NEXT_INTERVALSINT)

DECLAREOLDER_THAN_PARTITION_DATEVARCHAR(16);

DECLAREPARTITION_NAMEVARCHAR(16);

DECLARELESS_THAN_TIMESTAMPINT;

DECLARECUR_TIMEINT;

Untilhere,wehavedeclaredthevariableweneedafter.Now,onthenextline,wewillcallthestoredprocedureresponsibletocheckwhetherapartitionisalreadypresentandifnot,wewillcreatethem:

CALLpartition_verify(SCHEMA_NAME,TABLE_NAME,HOURLY_INTERVAL);

SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(NOW(),'%Y-%m-%d

00:00:00'));

www.it-ebooks.info

IFDATE(NOW())='2014-04-01'THEN

SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD(NOW(),

INTERVAL1DAY),'%Y-%m-%d00:00:00'));

ENDIF;

SET@__interval=1;

create_loop:LOOP

IF@__interval>CREATE_NEXT_INTERVALSTHEN

LEAVEcreate_loop;

ENDIF;

SETLESS_THAN_TIMESTAMP=CUR_TIME+(HOURLY_INTERVAL*@__interval

*3600);

SETPARTITION_NAME=FROM_UNIXTIME(CUR_TIME+HOURLY_INTERVAL*

(@__interval-1)*3600,'p%Y%m%d%H00');

Nowthatwehavecalculatedalltheparametersneededbythecreate_partitionprocedure,wecanrunit.Thisstoredprocedurewillcreatethenewpartitiononthedefinedschema:

CALLpartition_create(SCHEMA_NAME,TABLE_NAME,

PARTITION_NAME,LESS_THAN_TIMESTAMP);

SET@__interval=@__interval+1;

ENDLOOP;

SETOLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(),INTERVAL

KEEP_DATA_DAYSDAY),'%Y%m%d0000');

Thesectionthatfollowsisresponsibletoremovetheolderpartitions,usingtheOLDER_TAN_PARTITION_DATEprocedure,whichwehavecalculatedonthelinesbefore:

CALLpartition_drop(SCHEMA_NAME,TABLE_NAME,

OLDER_THAN_PARTITION_DATE);

DELIMITER;

Thisstoredprocedurewillbethecoreofourhousekeeping.Itwillbecalledwiththefollowingsyntax:

CALLpartition_maintenance('<zabbix_db_name>','<table_name>',

<days_to_keep_data>,<hourly_interval>,<num_future_intervals_to_create>)

Theprocedureworksbasedon1hourintervals.Next,ifyouwanttopartitiononadailybasis,theintervalwillbe24hours.Instead,ifyouwant1hourpartitioning,theintervalwillbe1.

Youneedtospecifythenumberofintervalsthatyouwantcreatedinadvance.Forexample,ifyouwant2weeksintervaloffuturepartitions,use14.Ifyourintervalis1(forhourlypartitioning),thenthenumberofintervalstocreateis336(24*14).

Thisstoredprocedureusessomeotherstoresprocedures:

partition_create:Thiscreatesthepartitionforthespecifiedtablepartition_verify:Thischeckswhetherthepartitionisenabledonatable,ifnot,thencreateasinglepartitionpartition_drop:Thisdropspartitionsolderthanatimestamp

Forallthedetailsaboutthesestoredprocedures,seeAppendixA,PartitioningtheZabbix

www.it-ebooks.info

Database.

Onceyou’vecreatedalltherequiredstoredprocedures,youneedtochangetwoindexestoenabletheminordertobereadyforapartitionedtable:

mysql>Altertablehistory_textdropprimarykey,addindex(id),drop

indexhistory_text_2,addindexhistory_text_2(itemid,id);

Records:0Duplicates:0Warnings:0

mysql>Altertablehistory_logdropprimarykey,addindex(id),dropindex

history_log_2,addindexhistory_log_2(itemid,id);

Records:0Duplicates:0Warnings:0

Oncethisisdone,youneedtoschedulethepartition_maintenance_allstoredprocedurewithacronjob.Formoredetailsaboutthepartition_maintenance_allprocedure,pleasechecktheinstructionscontainedinAppendixA,PartitioningtheZabbixDatabase.Thecronjobneedstoexecutethefollowingcommand:

mysql-h<zabbix_db_host>-u<zabbixuser>-p<zabbixpassword>zabbixdatabase

-e"CALLpartition_maintenance_all('zabbix');"

Oncethishasbeenset,youneedtobearinmindtodisablethehousekeepingforhistoryandtrends.VerifythattheOverrideitem<trend/history>periodZabbixconfigurationischeckedforbothhistoryandtrends.Here,youneedtosettheDatastorageperiod(indays)boxforhistoryandtrendstothevalueyou’vedefinedinyourprocedure,ourexampleinAppendixA,PartitioningtheZabbixDatabaseisof28and730.

www.it-ebooks.info

InstallingaZabbixproxyInstallationoftheZabbixproxyfrompackagesisaquitesimpletask.Onceyou’veaddedtheZabbixrepository,youonlyneedtorunthefollowingcommand:

$yuminstallzabbix-proxy-mysql

Thiswillinstalltherequiredpackages:

Installation:

zabbix-proxy-mysqlx86_642.4.0-1.el6zabbix390k

Installingfordependencies:

zabbix-proxyx86_642.4.0-1.el6zabbix21k

TheZabbixproxyinstallationisquitesimilartotheserverone.Onceyou’veinstalledtheserver,youneedtoinstallMySQL,createthedatabase,andimporttheDBschema:

$mysql-uroot-p

$mysql>CREATEDATABASEzabbixCHARACTERSETUTF8;

QueryOK,1rowaffected(0.00sec)

$mysql>GRANTALLPRIVILEGESonzabbix.*to'zabbixuser'@'localhost'

IDENTIFIEDBY'zabbixpassword';

$mysql>FLUSHPRIVILEGES;

$mysql>quit

Next,weneedtorestorethedefaultZabbixMySQLdatabasefiles:

proxy-mysql-2.4.0/create/schema.sql

Now,weneedtostartthedatabase,configuretheproxy,andstarttheservice.Inthisexample,wehaveconsideredtouseaZabbixproxythatreliesonaMySQLwithInnoDBdatabase.Thisproxycanbeperformedintwodifferentways:

Lightweight(andthenuseSQLite3)Robustandsolid(andthenuseMySQL)

Here,wehavechosenthesecondoption.Inalargenetworkenvironmentwheretheproxy,incaseofissue,needstopreserveallthemetricsacquireduntiltheserveracquiresthemetrics,it’sbettertoreduce,attheminimum,theriskofdataloss.Also,ifyouconsiderthisscenarioinalargenetworkenvironment,youmostlikelywillhavethousandsofsubnetworksconnectedtotheZabbixserverwithallthepossiblenetworkdevicesin-between.Well,exactly,thisisnecessarytouseadatabasethatcanpreventanydatacorruptions.

www.it-ebooks.info

InstallingtheWebGUIinterfaceTheWebGUIinterfacewillbedoneoncemoreusingtheRPMs.

Toinstallthewebinterface,youneedtorunthefollowingcommand:

$yuminstallzabbix-web-mysql

Yumwilltakecaretoresolveallthedependencies.Onceyou’redone,theprocessofthiscomponentisquiteeasy:weneedtoopenawebbrowser,pointatthefollowingURL:http://your-web-server/zabbix,andfollowtheinstructions.

OnthestandardRedHatsystem,yousimplyneedtochangetheseparametersonyour/etc/php.inifile:

php_valuemax_execution_time300

php_valuememory_limit128M

php_valuepost_max_size16M

php_valueupload_max_filesize2M

php_valuemax_input_time300

Also,setyourtimezoneonthesamefile(forexample,php_valuedate.timezoneEurope/Rome).

Now,it’stimetostartupApache,butbeforethis,weneedtocheckwhetherwehaveSELinuxenabledandonwhichmode?TocheckyourSELinuxstatus,youcanrun:

#sestatus

SELinuxstatus:enabled

SELinuxfsmount:/selinux

Currentmode:permissive

Modefromconfigfile:permissive

Policyversion:24

Policyfromconfigfile:targeted

Now,youneedtocheckwhetheryouhavethehttpddaemonenabledtousethenetworkwiththefollowingcommand:

#getseboolhttpd_can_network_connect

httpd_can_network_connect-->off

Mostlikely,youwillhavethesamekindofresult,thenallweneedtodoisenablethehttpd_can_network_connectoptionusingthenextcommandwith–Ptopreservethevalueafterareboot:

#setsebool–Phttpd_can_network_connecton

#getseboolhttpd_can_network_connect

httpd_can_network_connect-->on

Now,allthatwestillhavetodoisenablethehttpddaemonandstartourhttpdserver:

#servicehttpdstart

Startinghttpd:[OK]

Next,enablethehttpdserverasaservice:

www.it-ebooks.info

#chkconfighttpdon

Wecancheckthechangedonewiththenextcommand:

#chkconfig--listhttpd

httpd0:off1:off2:on3:on4:on5:on6:off

Onceyou’vedonethis,youonlyneedtofollowthewizard,andinafewclicks,youwillhaveyourwebinterfacereadytostartup.

TipIfyouknowthattheloadagainstthewebserverwillbehigh,duetoahighnumberofaccountsthatwillaccessit,probably,it’sbettertoconsiderusingNginx.

Now,youcanfinallystartyourZabbixserverandthefirstentryinthe/var/log/zabbix/zabbix_server.logfilewilllooksomethinglikethefollowingcode:

37909:20140925:091128.868StartingZabbixServer.Zabbix2.4.0(revision

48953).

37909:20140925:091128.868******Enabledfeatures******

37909:20140925:091128.868SNMPmonitoring:YES

37909:20140925:091128.868IPMImonitoring:YES

37909:20140925:091128.868WEBmonitoring:YES

37909:20140925:091128.868VMwaremonitoring:YES

37909:20140925:091128.868Jabbernotifications:YES

37909:20140925:091128.868EzTextingnotifications:YES

37909:20140925:091128.868ODBC:YES

37909:20140925:091128.868SSH2support:YES

37909:20140925:091128.868IPv6support:YES

37909:20140925:091128.868******************************

37909:20140925:091128.868usingconfigurationfile:

/etc/zabbix/zabbix_server.conf******************************

Next,youcanstarttoimplementandacquirealltheitemscriticalforyournetwork.

www.it-ebooks.info

SummaryInthischapter,wecoveredalargenumberofcomponents.Westartedwithdefiningwhatalargeenvironmentis.Wealsosawhowthenetworksetupcanbedesignedandhowitcanevolvewithinyourinfrastructure.Wesawtheheaviesttaskontheserverside(housekeeping)andhowtoavoidperformancedegradationduetothis.WediscussedMySQLpartitioningin-depth.Wealsobrieflydiscussedthedifferencesbetweenactiveandpassiveproxies;youwillnowbeabletodecidehowtosetthemupandwhichonetochooseonceyouknowyournetworktopology.Also,wesawhowtoacquiresomecriticalmetricstomonitortheZabbixproxyconnectionandtheamountofitemsthatitstillneedstosendus.

Asyoucansee,wecoveredalotofargumentsinjustonechapter;wedidthisbecausewewouldliketousemorespaceintheupcomingchapters.Inthenextchapter,wewillexplorethedifferentappliancesandprotocolsatlayer2andlayer3oftheISO/OSIstack.Also,youwillseehowtobestextrapolatemeaningfulmonitoringdatafromthecollectedmeasurefortheprotocollayers2and3.

www.it-ebooks.info

Chapter2.ActiveMonitoringofYourDevicesNowthatyouhaveaworkingZabbixsetup,it’stimetotakealookatyournetworkandfigureoutthecomponentsthatyouwanttomonitor,thekindofdatayouwanttocollect,andtheconditionsunderwhichyouwanttobenotifiedaboutproblemsandstatechanges.

Itwouldbeimpossibleforanybookonthistopictofullycoverallthedifferentkindsofnetworkappliancesandtopologiesandallthedifferentmonitoringscenariosthatanetworkadministratormightneedaseveryenvironmenthasitsownspecificquirksthatagoodmonitoringsolutionhastoaccountfor.ThischapterwillofferyouafewexamplesofthedifferentmonitoringpossibilitiesZabbixcanachievebyrelyingondifferentmethodsandprotocols.You’llseehowtoqueryyournetworkfromthedatalinklayeruptoroutingandnetworkflowusingICMP,SNMP,andlogparsingfacilitiestocollectyourmeasurements.

You’lllearnhowtoextractmeaningfulinformationfromthedatayougatheredusingaggregatedandcalculateditemsandhowtoconfigurecomplextriggersthatwillalertyouaboutrealnetworkissueswhileminimizinguninterestingornonrelevantdata.

Bytheendofthechapter,you’llhaveagoodoverviewofZabbix’snetworkmonitoringpossibilities,andyou’llbereadytoadaptwhatyoulearnedforyourspecificrequirements.Butlet’sfirsthaveaquickoverviewofhowZabbixorganizesmonitoringdatawithhosts,templates,items,andtriggers.

www.it-ebooks.info

UnderstandingZabbixhostsOneofZabbix’sgreatstrengthsisitsflexibilitywhenitcomestoorganizingmonitoringdata.Evenwithoutconsideringitspowerfultemplatinganddiscoveryfeatures,whichwillbecoveredinChapter4,DiscoveringYourNetwork,thereisalotthatyoucandowithstandardhosts,items,andtriggers.Hereareafewtipsonhowyoucanusethemeffectively.

www.it-ebooks.info

HostsandhostgroupsZabbixhostsusuallyrepresentasingle,specificboxorapplianceinyournetwork.Theycanalsobeapartofoneormorehostgroups.

HostgroupsareveryusefulastheymakeiteasytonavigateZabbix’sinterface,separatinghostsintocategoriesandallowingyoutoorganizeandmanageahugeamountofapplianceswithouthavingtodealwithimpossiblylonglistsofhostnames.Thesamehostcanbepartofdifferenthostgroups,andthiscanbeveryusefulasyoumightwant,forexample,tohaveagroupforallyourrouters,agroupforallyourswitches,andagroupforeverysubnetyoumanage.So,asinglerouterwillbepartoftheroutersgroupandallthesubnetgroupsithasaninterfaceon,whileaswitchwillbepartoftheswitchesgroupandofthesubnetit’spartof,andsoon.

Whilethisiscertainlyagoodwaytoorganizeyourhosts,bothtovisualizeandtomanageyourmonitoringdata,thereareacoupleofnot-too-obviouspitfallsyoushouldbeawareofifyoudecidetoputthesamehostinmultiplegroups:

Calculateditemsshowaggregatemonitoringdatabasedonhostgroupmembership.Ifyouconfigureanaggregateditemthatusesmorethanonecalculateditemfromdifferenthostgroups,youcanendupusingthesamehost’sdatamorethanonce,introducingasignificanterrorinyourcalculations.Actionsareusuallyfilteredbasedonhostgroups.Thismeansthatthesametriggereventcouldfireupmorethanoneactionifthehostispartofmorethanonehostgroup,leadingtopotentiallyduplicatemessagesandalerts.Useraccesspermissionsarehost-group-based.Thismeansthatsomeuserscouldbeabletoseemorehostsandmonitoringdatathantheyactuallyneedtoifahostendsupinahostgrouptheyhaveaccessto.

Thisisbynomeansanattempttodiscouragethepracticeofassigningmultiplehostgroupstothesamehost.Justbeawareoftheramificationsofsuchapracticeanddon’tforgettotakeintoconsiderationtheaddedcomplexitywhenyouconfigureyouritems,actions,andaccesspermissions.

HostinterfacesEachhostiscomposedofacollectionofitemsthatrepresenttherawmonitoringdata,andtriggers,whichrepresentZabbix’smonitoringintelligencebasedonthedatagathered.It’salsocomposedofaseriesofinterfacesthattelltheZabbixserverorproxyhowtocontactthehosttocollecttheaforesaidmonitoringdata.Mostnetworkapplianceshavemorethanoneinterface,soyouwouldwanttomakesurethatallhoststhatrepresentrouters,firewalls,proxies,gateways,andwhatnot,arelistingallthoseappliances’interfacesandtheiraddresses.Theadvantagesareobvious:

You’llbeabletoquicklyreviewwhataddressesareconfiguredonaspecifichostwhilelookingatmonitoringdataYou’llbeabletodifferentiateyourchecksbyqueryingdifferentaddressesorportsofthesamehostbasedonyourneeds

www.it-ebooks.info

Yourmapsandtopologieswillbemoreconsistentwithwhat’sactuallydeployed

Addinginterfacestoahostisfairlystraightforward.AllyouneedtodoisnavigatetoConfiguration|Hostsandthenselectthehostyouwanttoedit.Theinterfacessectionisinthemainconfigurationtab,asshowninthefollowingscreenshot:

Asyoucanseeintheaboveexample,therearethreeagentinterfacesthatshowallthenetworkstherouterisconnectedtoandjustoneSNMPinterface.AgentinterfacesareusednotonlyforZabbixagentitems,butalsoforsimpleandexternalchecks.Ontheotherhand,you’lluseSNMPinterfacestosendSNMPqueriestoyourhost.Theprecedingexampleassumesthatyou’llonlyuseSNMPontherouter’sinterfacethatisconnectedtoamanagementnetwork(192.168.1.0inthisexample),whileyou’llalsouseICMP,TCP,andexternalchecksonitstwoproductioninterfaces.Ofcourse,youarefreetoconfiguredifferentIPaddressesforAgentandSNMPinterfacesdependingonwhatprotocolsandchecksyouplantoactivateonwhichinterfaces.

HostinventoryHavinginventorydatadirectlyavailableinyourmonitoringsolutionhasalotofobviousadvantageswhenitcomestoattachingusefulinformationtoyouralertsandalarms.Unfortunately,themorehostsyouhavetomanage,themoreessentialitistohaveup-to-dateinventoryinformation,andtheharderitistomaintaintheaforesaidinformationinareliableandtimelymanner.Manuallyupdatingahost’sinventorydatacanquicklybecomeanimpossibletaskwhenyouhavetensorhundredsofhoststomanage,andit’snotalwayspossibletowriteautomatedscriptsthatwilldothejobforyou.Fortunately,Zabbixoffers

www.it-ebooks.info

anautomaticinventoryfeaturethatcanatleastpartiallyfillininventorydatabasedonactualmonitoringdata.Toactivatethisfeature,firstyou’llneedtoselectAutomaticintheHostinventorytabofahostconfigurationpageandthenmovetotheitemsthatyou’llusetopopulatetheinventorydata.

Whenconfiguringanitem,youshouldassignitsdatatoaspecificinventoryfieldsothattheaforesaidfield’svaluewillbesetandautomaticallyupdatedbasedontheitem’smeasurements,asshowninthefollowingscreenshot:

Asyoucanseeintheprecedingexample,ahost’slocationinventoryvaluewillbepopulatedbasedonthecorrespondingSNMPquery.Thismeansthatifyouchangeadevice’slocationinformation,thatchangewillbereflectedinZabbixassoonastheitem’svalueispolledonthedevice.Dependingonthedataavailableonthedevice,you’llbeabletopopulateonlyafewinventoryfieldsormostofthem,whilefallingbackonmanualupdatesofthefieldsthatfalloutsideofyourdevice’sreportingpossibilities.

Speakingofitems,let’snowfocusonthedifferentmonitoringpossibilitiesthatZabbixitemsofferandhowtoapplythemtoyourenvironment.

www.it-ebooks.info

GoingbeyondZabbixagentsTherearecertainlymanyadvantagesinusingZabbix’sownagentsandprotocolwhenitcomestomonitoringWindowsandUnixoperatingsystemsortheapplicationsthatrunonthem.However,whenitcomestonetworkmonitoring,thevastmajorityofmonitoredobjectsarenetworkappliancesofvariouskinds,whereit’softenimpossibletoinstallandrunadedicatedagentofanytype.Thisbynomeansimpliesthatyou’llbeunabletofullyleverageZabbix’spowertomonitoryournetwork.Whetherit’sasimpleICMPechorequest,anSNMPquery,anSNMPtrap,netflowlogging,oracustomscript,therearemanypossibilitiestoextractmeaningfuldatafromyournetwork.Thissectionwillshowyouhowtosetupthesedifferentmethodsofgatheringdata,andgiveyouafewexamplesonhowtousethem.

www.it-ebooks.info

SimplechecksLet’sstartwiththesimplestcase.Atfirstglance,simplechecksdon’tlookthatinteresting:excludingalltheVMwareHypervisorchecksthatareincludedinthiscategory,simplechecksarereducedtoacoupleofgenericTCP/IPconnectionchecksandthreeICMPechochecks,asfollows:

Checkname Description

Icmpping Thisreturns1ifthehostrespondstoanICMPping;0otherwise

Icmppingloss ThisreturnsthepercentageoflostICMPpingpackets

Icmppingsec ThisreturnstheICMPresponsetimeinseconds

Net.tcp.service Thisreturns1ifthehostacceptsconnectionsonaspecifiedTCPport;0otherwise

Net.tcp.service.perf ThisreturnsthenumberofsecondsspenttoobtainaconnectiononaspecifiedTCPport

Generallyspeaking,thesechecksprovemoreusefulasthedistancebetweenthemonitoringprobeandthemonitoredhostincreases,bothintermsofphysicaldistance(ageographicallinktoanothercityforexample)andintermsofhopsapackethastogothrough.Thismeansthatifyouareinterestedinyournetwork’sperformance,itwouldmakesensetoassignhostswithsimplecheckstoZabbixproxiesthatarenotinthesamesubnet,butaresituatedwheretheywillmimicascloselyaspossibleyouractualnetworktraffic.Net.tcp.serviceisparticularlyusefulfromthispointofview,notjusttocheckthestatusoftheavailabilityofspecificserviceswhenyoucannotuseZabbixagents,butalsotocheckgeneralhostavailabilityacrossrestrictivefirewallsthatblockICMPtraffic.

TipInordertoreducenetworktrafficandtomakemoreefficientICMPchecks,Zabbixusesfpinginsteadoftheregularpingwhenexecutingicmpping,icmppingloss,andicmppingsecitemchecks.

MakesureyouhavefpinginstalledonyourZabbixserverandalsoonalltheZabbixproxiesthatmightneedit.Ifyoudon’thaveit,asimpleyuminstallfpingwillusuallybeenoughfortheZabbixdaemonstofinditanduseit.

Whilebothnet.tcp.serviceandnet.tcp.service.perfdosupportsomewell-knownprotocols,suchasSSH,FTP,HTTP,andsoon,thesetwoitems’mostusefuloptionisprobablytheonethatallowsyoutoperformasimpleTCPhandshakeconnectionandcheckwhetheraspecificIPisreachableonaspecificport.Thesekindofchecksareusefulbecause,justlikeICMPpings,theywillmostlyinvolvethenetworkstack,reducingapplicationoverheadtoaminimum,thusgivingyoudatathatmorecloselymatchesyouractualnetworkperformance.Ontheotherhand,unlikeICMPpings,theywillallowyoutocheckforTCPportavailabilityforagivenhost.Obvioususecasesincludemakinglightweightservicechecksthatwillnotimpactverybusyhostsorappliancestoomuch,

www.it-ebooks.info

andmakingsurethatagivenfirewallisallowingtrafficthrough.

Aslightlylessobvioususecaseisusingoneormorenet.tcp.serviceitemstomakesurethatsomeservicesarenotrunningonagiveninterface.Takeforexample,thecaseofaborderrouterorfirewall.Unlessyouhavesomeveryspecialandspecificneeds,you’lltypicallywanttomakesurethatnoadminconsolesareavailableontheexternalinterfaces.Youmighthavedouble-checkedtheappliance’sinitialconfiguration,butasystemupdate,acarelessadmin,orasecuritybugmightchangetheaforesaidconfigurationandopenyourappliance’sadmininterfacestoafarwideraudiencethanintended.AsecuritybreachlikethisonecouldpassunobservedforalongtimeunlessyouconfigureafewsimpleTCP/IPchecksonyourappliance’sexternalinterfacesandthensetupsometriggersthatwillreportaproblemifthosechecksreportanopenandresponsiveport.

Let’staketheexampleoftherouterwithtwoproductioninterfacesandamanagementinterfaceshowninthesectionabouthostinterfaces.Iftherouter’sHTTPSadminconsoleisavailableonTCPport8000,you’llwanttoconfigureasimplecheckitemforeveryinterface:

Itemname Itemkey

management_https_console net.tcp.service[https,192.168.1.254,8000]

zoneA_https_console net.tcp.service[https,10.10.1.254,8000]

zoneB_https_console net.tcp.service[https,172.16.7.254,8000]

Allthesecheckswillreturn1iftheserviceisavailable,and0iftheserviceisnotavailable.Whatchangesishowyouimplementthetriggersontheseitems.Forthemanagementitem,you’llhaveaproblemiftheserviceisnotavailable,whilefortheothertwo,you’llhaveaproblemiftheserviceisindeedavailable,asshowninthefollowingtable:

Triggername Triggerexpression

Managementconsoledown {it-1759-r1:net.tcp.service[http,192.168.1.254,8000].last()}=0

ConsoleavailablefromzoneA {it-1759-r1:net.tcp.service[http,10.10.1.254,8000].last()}=1

ConsoleavailablefromzoneB {it-1759-r1:net.tcp.service[http,172.16.7.254,8000].last()}=1

Thisway,you’llalwaysbeabletomakesurethatyourdevice’sconfigurationwhenitcomestoopenorclosedportswillalwaysmatchyourexpectedsetupandbenotifiedwhenitdivergesfromthestandardyouset.

Tosummarize,simplechecksaregreatforallcaseswhereyoudon’tneedcomplexmonitoringdatafromyournetworkastheyarequitefastandlightweight.Forthesamereason,theycouldbethepreferredsolutionifyouhavetomonitoravailabilityforhundredstothousandsofhostsastheywillimpartarelativelylowoverheadonyour

www.it-ebooks.info

overallnetworktraffic.

Whenyoudoneedmorestructureandmoredetailinyourmonitoringdata,it’stimetomovetothebreadandbutterofallnetworkmonitoringsolutions:SNMP.

www.it-ebooks.info

KeepingSNMPsimpleTheSimpleNetworkMonitoringProtocol(SNMP)isanexcellent,generalpurposeprotocolthathasbecomewidelyusedbeyonditsoriginalpurpose.Whenitcomestonetworkmonitoringthough,it’salsooftentheonlyprotocolsupportedbymanyappliances,soit’softenaforced,albeitnaturalandsensible,choicetointegrateitintoyourmonitoringscenarios.Asanetworkadministrator,youprobablyalreadyknowallthereistoknowaboutSNMPandhowitworks,solet’sfocusonhowit’sintegratedintoZabbixandwhatyoucandowithit.

Firstofall,we’llneedtotalkaboutSNMPgetsandSNMPtrapsintwodifferentdiscussionsastheyareimplementedandusedindifferentwaysbyZabbix.ThereasonforthisseparationisintheverynatureofSNMPgetsasopposedtoSNMPtraps.AnSNMPgetrepresentsasingle,discretepieceofinformationthatrepresentsthecurrentstatusofametric,andit’snottiedtoanyspecificevent.Whetherit’sacounterwiththetotalnumberofbytesthatpassedthroughaninterface,aBooleanvaluethatwilltellifalinkisupordown,orastringwithanappliance’slocationorcontactinformation,anSNMPvaluewillbeavailableatanymoment,anditwillbepossibletopollitwithanarbitraryfrequency.

ThismapsnicelytoZabbixitems.JustlikeSNMPgetvalues,theyalsorepresentsingle,discretevaluesthatcanbepolledwitharbitraryfrequency.ThismakesitreallystraightforwardtouseregularSNMPqueriestopopulateZabbixitemssincetheonlythingsyouhavetoworryaboutaretheSNMPOID,thedatatype,andthecommunitystringorauthenticationinformation.We’llseeafewexamplesinthenextparagraph.

AnSNMPtraprepresentsaspecificeventthathappensataspecificpointintime.Itmightrepresentalinkstatechange,arebootevent,orauserlogin.Inanycase,youcannotquerythestateofanSNMPtrap;youjusthavetowaittoreceiveone,anditwillnotrepresentasingle,discretevaluebutachangefromonevaluetoanother.Theyresemble,inmanyways,Zabbixeventsinsteadofrawdata.ThiscomplicatesthingsalittlesinceZabbixeventsaretheresultofevaluatingtriggersagainstcollecteddata,whileSNMPtrapscanonlyenterZabbixasitemvalues,thatis,ascollecteddata.Sowe’llneedtoresolvethisapparentmismatchinordertofullyleveragetheinformationcontainedinSNMPtraps.We’llseehowinashortwhile,butfirstlet’slookatafewdetailsconcerningregularSNMPqueriesexecutedfromZabbix.

GettingSNMPdataintoZabbixAZabbixserverusuallycomeswithgoodSNMPsupportoutofthebox.Notonlydoesitsupportthequeryingprotocolnatively,butitalsocomesequippedwithanumberofSNMPtemplatesthatcangetyoustartedintherightdirection.ThismeansthatformostdevicesyouonlyhavetolinktheTemplateSNMPDevicetemplate,andyou’llimmediatelybeabletogetsomebasicinformationaboutit,asshowninthefollowingscreenshot:

www.it-ebooks.info

We’vealreadyseenhowtheDevicelocationitemcanbeusedtopopulateahost’sinventorylocationrecord,butthereareacoupleofotherusefulbitsofinformationintheabovepicture.

Firstofall,there’salow-leveldiscoveryruletoexplore.We’lldelvemoredeeplyintodiscoveryrulesinChapter4,DiscoveringYourNetwork,butfornow,we’lljustseethatit’saboutdynamicallycreatingnetworkinterfaceitems:

Foreveryinterface,eightitemswillbecreated,includingtheinterfacename,operationalstatus,incomingandoutgoingtraffic,andsoon.Thismeansthatthesametemplatewillbeusefulforthebasicmonitoringofnetworkapplianceswithanynumberofnetworkinterfaces.

Thesecondthingtonotice,lookingatbothimages,istheupdateinterval,andhistoryandtrendretentionperiodsfortheitems.Zabbixtriestosetsomesensibledefaults,butyou’llprobablyneedtoupdatesomeofthosevaluesbasedonthenumberofmonitoredhostsyouhaveinyourenvironment,yourstoragespaceavailability,andthenetworkloadofyourmonitoringtraffic.

NoteAnotherparameterthatisrelatedtoZabbix’sperformanceistheinitial(andminimum)numberofpollersthattheserverkeepsactiveatanygiventime.Ifyoufindthatyourpollingqueueisgettinglonger,youmightwanttoincreasethenumberofpollersinzabbix_server.conf.Theavailabledefaultoptionsare:

www.it-ebooks.info

#StartPollers=5

#StartIPMIPollers=0

#StartPollersUnreachable=1

#StartTrappers=5

#StartPingers=1

#StartDiscoverers=1

#StartHTTPPollers=1

Workyourwayupslowly,oryou’lljustendupwithunnecessaryprocessesbeingcreatedwhenZabbixisstarted.

Ifyouhavehundredsofhoststomonitor,andforeveryhost,youcollecttensofsinglemeasurementseveryminute,youwouldreachapointwhereyourZabbixserver’snetworkloadorCPUloadwillstarttoimpactontheserver’sperformance,leadingtodelaysinitempollingordroppedconnections.Ifyoucannotjustupgradetomorepowerfulhardware,youmighthavetotweakthepollingintervalofyourtemplatessothattheystrikeagoodbalancebetweengranularityofdetailandperformance.

Adevice’sname,contactdetails,description,location,andsuchlike,willrarelychangeoncethedevicehasbeendeployed,soitwouldbeawastetopollforthosevalueseveryhour(3,600seconds).Bychangingtheintervalto6hoursorevenaday,you’llautomaticallyreduceyournetworktrafficrelatedtoessentiallyfixedinformationbyafactorof6,upto24.

Raisingthepollingintervalforsomeoftheinterfacecounterscanhaveanevenmoredramaticimpactonyoursystemandnetworkload.Whileyou’llprobablywanttochecktheadminandoperationalstatusofaninterfaceasoftenaspossible—otherwiseyouruntheriskofnotgettingnotifiedaboutpossibleproblemsinatimelymanner—ontheotherhand,you’llprobablybeabletolivewithpollingincomingandoutgoingtrafficanderrorseveryfiveminutes(300seconds)insteadofeveryminute.Yourgraphswillstillbeverydetailed,butyournetworkwillbemuchlessfloodedwithSNMPrequests.Keepinmindthatchangeslikethesemightnotseemmuchwhenreferredtoasinglehost,butasthenumberofyourmonitoredobjectsgrow,youcanveryquicklyrunuptohundredsoreventhousandsofnewmonitoringvaluespersecondcomingintoyourZabbixserver.

Thesamecanbesaidwhenitcomestoretentionperiodsandstoragespace.Inthiscase,keepinmindthattrendsstoreaboutthreevaluesperhour(min,maxandaverage)overthetimerangespecified,whilehistorystoresallvaluescollectedinthespecifiedtimerange.Thismeansthatbasedonyourpollinginterval,it’susuallycheapertoextendatrendretentionvaluethanahistoryone.Thisis,ofcourse,validonlyfornumericalvaluesasstringonescan’treallyhavetrends,justhistory.

OnelastthingtonoticeintheaboveimagesisthatthemonitoringprotocolforallitemsissettoSNMPv2.JustlikeSNMPv1,SNMPv2doesn’tofferrealsecurityforthemonitoringdatathatcrossesthenetworkbetweenanapplianceandthemonitoringserver:alltrafficissentandreceivedintheclear,andtheSNMPcommunityisjustastring,easilyparsablefrominterceptedtraffic.Whileit’scertainlytruethatafewnetworkappliancesdon’tsupportSNMPv3becauseeithertheyaretoooldortheyaretoosimple,It’salsotruethat

www.it-ebooks.info

thenewversionoftheprotocolhasbeenaroundforquiteawhilenowandanumberofappliancesdosupportit.ThemainadvantagesofSNMPv3areitsauthenticationandencryptioncapabilities.Thesecanhelpmakesurethatallmonitoringtrafficisnotbogusorcorrupted,andthatit’skeptconfidentialfrompryingeyes.Thisisparticularlyimportantifyouneedtomonitorsomehostsoveranetworklinkyouhavenorealcontrolover,suchasaWANconnectionthroughathird-partyprovider.ItwouldalwaysbenicetouseSNMPv3acrossyournetwork,butincaseslikethese,youarestronglyencouragedtodosoasthere’sarealpossibilitythatyourtrafficcanbeindeedinterceptedandtappedinto.

Let’staketheexampleofaCiscorouter,andlet’sseehowtoconfigureSNMPv3onitbeforemovingontotheZabbixside.

Firstofall,let’screateamonitoringgroup.Thisisusedtodefineaccesstothedevice’sMIBs.OntheCiscorouter,openaconsolesessionandgointoconfigurationmode.Thenissuethefollowingcommand:

R1(config)#snmp-servergroupMonitoringGroupv3priv

Thev3keywordspecifiesthatwewanttouseSNMPv3,whiletheprivkeywordspecifiesthatwewanttousebothauthenticationandencryption.It’spossibletopassmoreoptionstotheprecedingcommandinordertodefineanaccesslistifyouwanttolimitaccesstospecificMiBs,butwe’llkeepthingssimplehereandletourZabbixprobeaccessallMIBs.

Nowthatwehaveagroup,wecancreateauser,asfollows:

R1(config)#snmp-serveruserzabbixMonitoringGroupv3authshazbxpasspriv

aes128zbxpriv

Asyoucansee,weassignedtheZabbixusertothepreviouslycreatedgroupanddefinedtheauthenticationandencryptionpassphrases.Takenoteofalltheseelementsasyou’llneedtospecifyallofthemonZabbix’ssideandtheywillneedtomatchwhatyouusedhere.Tosummarize,hereiswhatyou’llinputlaterwhenconfiguringanSNMPv3Zabbixitem:

Field Value

User zabbix

Authenticationprotocol sha

Authenticationpassphrase zbxpass

Privacyprotocol aes

Privacypassphrase zbxpriv

NotePleasedon’tusethepassphrasesshownhere.Theseareintentionallyweak,andweusedthemforillustrationpurposesonly.

Thisisallthereistoit.Later,we’lladdsomeinformationabouttellingtheappliance

www.it-ebooks.info

wheretosendSNMPtraps,butfornowyou’rereadytogetSNMPvaluesfromyourappliance,solet’sfocusonthatforawhile.

FindingtherightOIDstomonitorWhileZabbix’sdefaultSNMPtemplateswillhelpyougetstartedwithbasicmonitoring,you’llsoonfindtheneedtopollyourdevicesformoreinformation.Todothat,you’llneedtoknowtheOIDofthemetricyouwanttomonitoraswellasthedatatypeitwillyield.Afirstoptionistoconsultyourvendor’sdocumentationonthedeviceandfindoutwhichMIBsandOIDsareexposedbytheSNMPagent.Another,moreinteractive,optionistofindthemusingthesnmpwalkutilityanddirectlyaskingyourdeviceforthem.

NoteIfyoudon’talreadyhavesnmpwalk(andtheotherSNMPutilitiesforLinux)installed,youcanquicklydosowithasimplecommand:

#yuminstallnet-snmp-utils

OIDsaresentandreceivedbySNMPagentsandserversasdottedsequencesofnumbers.JustlikeIPaddresses,thisisconvenientformachine-to-machinecommunication,buthardtoreadforhumans.Inordertomakethemostfromtheexplorationofyourdeviceusingsnmpwalk,makesureyouhavealltheMIBsyouneedinstalled.MIBsessentiallymapOIDstoreadableandunderstandabledescriptionsofthemselves.Inotherwords,theytakeoutputlikethisone:

.1.3.6.1.2.1.2.2.1.1.1=INTEGER:1

.1.3.6.1.2.1.2.2.1.1.2=INTEGER:2

.1.3.6.1.2.1.2.2.1.1.3=INTEGER:3

.1.3.6.1.2.1.2.2.1.1.5=INTEGER:5

.1.3.6.1.2.1.2.2.1.2.1=STRING:lo

.1.3.6.1.2.1.2.2.1.2.2=STRING:eth1

.1.3.6.1.2.1.2.2.1.2.3=STRING:tap0

.1.3.6.1.2.1.2.2.1.2.5=STRING:br0

.1.3.6.1.2.1.2.2.1.3.1=INTEGER:softwareLoopback(24)

.1.3.6.1.2.1.2.2.1.3.2=INTEGER:ethernetCsmacd(6)

.1.3.6.1.2.1.2.2.1.4.1=INTEGER:16436

.1.3.6.1.2.1.2.2.1.4.2=INTEGER:1500

.1.3.6.1.2.1.2.2.1.4.3=INTEGER:1500

.1.3.6.1.2.1.2.2.1.4.5=INTEGER:1500

.1.3.6.1.2.1.2.2.1.5.1=Gauge32:10000000

.1.3.6.1.2.1.2.2.1.5.2=Gauge32:1000000000

.1.3.6.1.2.1.2.2.1.5.3=Gauge32:10000000

.1.3.6.1.2.1.2.2.1.5.5=Gauge32:0

.1.3.6.1.2.1.2.2.1.6.1=STRING:

.1.3.6.1.2.1.2.2.1.6.2=STRING:0:c:29:24:15:50

.1.3.6.1.2.1.2.2.1.6.3=STRING:2:10:f7:72:77:50

.1.3.6.1.2.1.2.2.1.6.5=STRING:0:c:29:24:15:50

.1.3.6.1.2.1.2.2.1.7.1=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.7.2=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.7.3=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.7.5=INTEGER:up(1)

www.it-ebooks.info

.1.3.6.1.2.1.2.2.1.8.1=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.8.2=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.8.3=INTEGER:up(1)

.1.3.6.1.2.1.2.2.1.8.5=INTEGER:up(1)

Then,theyturnitintoamuchmorereadableform:

IF-MIB::ifIndex.1=INTEGER:1

IF-MIB::ifDescr.1=STRING:lo

IF-MIB::ifDescr.2=STRING:eth1

IF-MIB::ifDescr.3=STRING:tap0

IF-MIB::ifDescr.5=STRING:br0

IF-MIB::ifType.1=INTEGER:softwareLoopback(24)

IF-MIB::ifType.2=INTEGER:ethernetCsmacd(6)

IF-MIB::ifMtu.1=INTEGER:16436

IF-MIB::ifSpeed.1=Gauge32:10000000

IF-MIB::ifPhysAddress.1=STRING:

IF-MIB::ifPhysAddress.2=STRING:0:c:29:24:15:50

IF-MIB::ifPhysAddress.3=STRING:2:10:f7:72:77:50

IF-MIB::ifPhysAddress.5=STRING:0:c:29:24:15:50

IF-MIB::ifAdminStatus.1=INTEGER:up(1)

IF-MIB::ifOperStatus.1=INTEGER:up(1)

IfyouhavetherightMIBs,youwon’thavetoguessthemeaningofeachOIDfromitsvalueasmostofthetime,itwillbeclearenoughfromitsname.ToaddanewMIBtoyourSNMPtools,youhavetoobtainitfromthevendorofyourdeviceandtheninstallitonyoursystem.VendorsusuallymaketheirMIBsfreelyavailable,soyoushouldn’thaveanyproblemsfindingthem.

HerearesomeofthemajorvendorsofMIBsources,compiledatthetimeofwriting:

Vendor MIBs

Cisco http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

Juniper http://www.juniper.net/techpubs/software/index_mibs.html

Barracudanetworks https://techlib.barracuda.com/search/go/global?q=MIB

www.it-ebooks.info

NoteAveryusefulresourceisOIDView’sfreeMIBdatabasethatyoucanfindhere:

http://www.oidview.com/mibs/detail.html

Atthetimeofwritingthis,thedatabasehadmorethan7,000MIBs,sochancesareyou’llbeabletofindaMIBforthemostobscurenetworkdeviceyoumighthavetomonitor.

MIBsareplaintextfiles,soifyouhaveacompressedarchive,youwillneedtounpackitbeforeyoucaninstallitscontents.OnceyouhavetheplaintextMIBS,it’sasimplematterofcopyingtheminto/usr/share/snmp/mibsandthenusingthe-moptiontotheSNMPcommandstospecifywhichMIByouwanttoloadinadditiontothedefaultones.

ShouldyourMIBscollectionbecometoobigandyouwantedtoorganizethemindifferentdirectories,thenyou’llneedtotellyourtoolswheretofindthem.Youhavetwooptions:eitherspecifyfromthecommandlinethedirectoriesyouwantyourcommandtosearchforMIBs,orputthisinformationinaconfigurationfilesothatyourcommandsalwaysknowtheMIBs’location.Theoptionsarediscussedasfollows:

Thefirstoptionisusefulifyou’rejusttryingoutanewMIBandseeingwhetherthat’stheoneyouneed.EveryNet-SNMP-basedcommandwilltakea-moptionthatyoucanusetospecifyaspecificMIBtoloadfromthemibsdirectory.Here’sacommandforexample:

$snmpwalk-m+CISCO-STUN-MIB-v3-uzabbix-aSHA-Azbxpassword-l

AuthPriv-xAES-Xprivpassword10.10.1.9

ThiscommandwilluseSNMPv3tocontacttheSNMPagentat10.10.1.9withthespecifiedcredentialsandwillloadtheCISCO-STUN-MIBthatitwillfindinthe/usr/share/snmp/mibsdirectory,inadditiontothosealreadyloadedasdefault.

Thesecondoptionismorepermanentandinvolvesediting(orcreating,ifit’snotalreadythere)the/etc/snmp/snmp.conffile.JustaddalinewiththelistofdirectoriestosearchformibsandanotherlinethatspecifieswhichMIBsthecommandsshouldactuallyload(inthiscase,we’llloadallofthem),asfollows:

mibdirs

/usr/share/snmp/mibs:/usr/share/snmp/mibs/cisco:/usr/share/snmp/mibs/ju

niper:/mnt/remote/shared_mibs/

mibs+ALL

Asyoucansee,evenifyoukeepyoursubdirectoriesin/usr/share/snmp/mibs,you’llhavetospecifyeachoneyouwantautomaticallyincluded.OnceyouhaveyourMIBsinstalledandloaded,you’llbereadytofullyexploreyourdevices’SNMPagents.ToperformacompletesnmpwalkonadevicecantakequitealotoftimeandproducealotofoutputdependingonhowmanyOIDsitexposes.Aroutercanhavethousandsofthem,soit’sadvisabletoredirectthecommand’soutputtoafilesothatyouareabletoreferenceitandexploreitatanytimeyouwantwithouthavingtoperformacompletewalkonthedeviceitself,asfollows:

$snmpwalk-v3-uzabbix-aSHA-Azbxpassword-lAuthPriv-xAES-X

www.it-ebooks.info

privpassword10.10.1.9>router-R1-snmp_baseline.txt

AnotheradvantageofhavingtheMIBsyouneedisthatit’llbeeasiertocreatenewSNMPitemsinZabbixasyou’llbeabletospecifythestringversionofanOIDandnotonlyitsnumericalvalue.ZabbixreliesontheNet-SNMPlibrary,soitwillalsoreferenceanyMIBsinstalledinyoursystem’sdefaultdirectories.

Solet’sseehowyoucanusetheoutputofsnmpwalktocreatenewZabbixitems.

MappingSNMPOIDstoZabbixitemsAnSNMPvalueiscomposedofthreedifferentparts:theOID,thedatatype,andthevalueitself.WhenyouusesnmpwalkorsnmpgettogetvaluesfromanSNMPagent,theoutputlookslikethis:

SNMPv2-MIB::sysObjectID.0=OID:CISCO-PRODUCTS-MIB::cisco3640

DISMAN-EVENT-MIB::sysUpTimeInstance=Timeticks:(83414)0:13:54.14

SNMPv2-MIB::sysContact.0=STRING:

SNMPv2-MIB::sysName.0=STRING:R1

SNMPv2-MIB::sysLocation.0=STRING:Upperfloorroom13

SNMPv2-MIB::sysServices.0=INTEGER:78

SNMPv2-MIB::sysORLastChange.0=Timeticks:(0)0:00:00.00

IF-MIB::ifPhysAddress.24=STRING:c4:1:22:4:f2:f

IF-MIB::ifPhysAddress.27=STRING:c4:1:1e:c8:0:0

IF-MIB::ifAdminStatus.2=INTEGER:down(2)

Andsoon.

Thefirstpart,theonebeforethe=signis,naturally,theOID.ThiswillgointotheSNMPOIDfieldintheZabbixitemcreationpageandistheuniqueidentifierforthemetricyouareinterestedin.SomeOIDsrepresentasingleanduniquemetricforthedevice,sotheyareeasytoidentifyandaddress.Intheaboveexcerpt,onesuchOIDisDISMAN-EVENT-MIB::sysUpTimeInstance.IfyouareinterestedinmonitoringthatOID,you’donlyhavetofillouttheitemcreationformwiththeOIDitselfandthendefineanitemname,adatatype,andaretentionpolicy,andyouarereadytostartmonitoringit.Inthecaseofanuptimevalue,time-ticksareexpressedinseconds,soyou’llchooseanumericdecimaldatatype.We’llseeinthenextsectionhowtochooseZabbixitemdatatypesandhowtostorevaluesbasedonSNMPdatatypes.You’llalsowanttostorethevalueasisandoptionallyspecifyaunitofmeasure.Thisisbecauseanuptimeisalreadyarelativevalueasitexpressesthetimeelapsedsinceadevice’slatestboot.Therewouldbenopointincalculatingafurtherdeltawhengettingthismeasurement.Finally,you’lldefineapollingintervalandchoosearetentionpolicy.Inthefollowingexample,thepollingintervalisshowntobe5minutes(300seconds),thehistoryretentionpolicyas3days,andthetrendstorageperiodasoneyear.Theseshouldbesensiblevaluesasyoudon’tnormallyneedtostorethedetailedhistoryofavaluethateitherresetstozero,or,bydefinition,growslinearlybyonetickeverysecond.

Thefollowingscreenshotencapsulateswhathasbeendiscussedinthisparagraph:

www.it-ebooks.info

Rememberthattheitem’skeyvaluestillhastobeuniqueatthehost/templatelevelasitwillbereferencedtobyallotherZabbixcomponents,fromcalculateditemstotriggers,maps,screens,andsoon.Don’tforgettoputtherightcredentialsforSNMPv3ifyouareusingthisversionoftheprotocol.

ManyofthemoreinterestingOIDs,though,areabitmorecomplex:multipleOIDscanberelatedtooneanotherbymeansofthesameindex.Let’slookatanothersnmpwalkoutputexcerpt:

IF-MIB::ifNumber.0=INTEGER:26

IF-MIB::ifDescr.1=STRING:FastEthernet0/0

www.it-ebooks.info

IF-MIB::ifDescr.2=STRING:Serial0/0

IF-MIB::ifDescr.3=STRING:FastEthernet0/1

IF-MIB::ifType.2=INTEGER:propPointToPointSerial(22)

IF-MIB::ifOperStatus.2=INTEGER:down(2)

IF-MIB::ifOperStatus.3=INTEGER:down(2)

IF-MIB::ifLastChange.1=Timeticks:(1738)0:00:17.38

IF-MIB::ifInOctets.1=Counter32:305255

IF-MIB::ifInDiscards.1=Counter32:0

IF-MIB::ifInErrors.1=Counter32:0

IF-MIB::ifOutOctets.1=Counter32:347968

Asyoucansee,foreverynetworkinterface,thereareseveralOIDs,eachonedetailingaspecificaspectoftheinterface:itsname,itstype,whetherit’supordown,theamountoftrafficcominginorgoingout,andsoon.ThedifferentOIDsarerelatedthroughtheirlastnumber,theactualindexoftheOID.Lookingattheprecedingexcerpt,weknowthatthedevicehas26interfaces,ofwhichweareshowingsomevaluesforjustthefirstthree.Bycorrelatingtheindexnumbers,wealsoknowthatinterface1iscalledFastEthernet0/0,itsMACaddressisc4:1:1e:c8:0:0,theinterfaceisupandhasbeenupforjust17

www.it-ebooks.info

seconds,andsometrafficalreadywentthroughit.

Now,onewaytomonitorseveralofthesemetricsforthesameinterfaceistomanuallycorrelatethesevalueswhencreatingtheitems,puttingthecompleteOIDintheSNMPOIDfield,andmakingsurethatboththeitemkeyanditsnamereflecttherightinterface.Thisprocessisnotonlypronetoerrorsduringthesetupphase,butitcouldalsointroducesomeinconsistenciesdowntheroad.Thereisnoguarantee,infact,thattheindexwillremainconsistentacrosshardwareorsoftwareupgradesorevenacrossconfigurationswhenitcomestomorevolatilestateslikethenumberofVLANsorroutingtablesinsteadofnetworkinterfaces.FortunatelyZabbixprovidesafeature,calleddynamicindexes,thatallowsyoutoactuallycorrelatedifferentOIDsinthesameSNMPOIDfieldsothatyoucandefineanindexbasedontheindexexposedbyanotherOID.

ThismeansthatifyouwanttoknowtheadminstatusofFastEthernet0/0,youdon’tneedtofindtheindexassociatedwithFastEthernet0/0(inthiscaseitwouldbe1)andthenaddthatindextoIF-MIB::ifAdminStatusofthebaseOID,hopingthatitwon’teverchangeinthefuture.Youcaninsteadusethefollowingcode:

IF-MIB::ifAdminStatus["index","IF-MIB::ifDescr","FastEthernet0/0"]

UponusingtheprecedingcodeintheSNMPOIDfieldofyouritem,theitemwilldynamicallyfindtheindexoftheIF-MIB::ifDescrOIDwherethevalueisFastEthernet0/0andappendittoIF-MIB::ifAdminStatusinordertogettherightstatusfortherightinterface.

Ifyouorganizeyouritemsthisway,you’llalwaysbesurethatrelateditemsactuallyshowtherightrelatedvaluesforthecomponentyouareinterestedinandnotthoseofanotheronebecausethingschangedonthedevice’ssidewithoutyourknowledge.Moreover,we’llbuildonthistechniquetodeveloplow-leveldiscoveryofadeviceaswe’llseeinChapter4,DiscoveringYourNetwork.

Youcanusethesametechniquetogetotherinterestinginformationoutofadevice.Consider,forexample,thefollowingexcerpt:

ENTITY-MIB::entPhysicalVendorType.1=OID:CISCO-ENTITY-VENDORTYPE-OID-

MIB::cevChassis3640

MIB::cevContainerSlot

MIB::cevCpu37452fe

ENTITY-MIB::entPhysicalClass.1=INTEGER:chassis(3)

ENTITY-MIB::entPhysicalClass.2=INTEGER:container(5)

ENTITY-MIB::entPhysicalClass.3=INTEGER:module(9)

ENTITY-MIB::entPhysicalName.1=STRING:3745chassis

ENTITY-MIB::entPhysicalName.2=STRING:3640ChassisSlot0

ENTITY-MIB::entPhysicalName.3=STRING:c3745MotherboardwithFast

EthernetonSlot0

ENTITY-MIB::entPhysicalHardwareRev.1=STRING:2.0

ENTITY-MIB::entPhysicalHardwareRev.2=STRING:

www.it-ebooks.info

ENTITY-MIB::entPhysicalHardwareRev.3=STRING:2.0

ENTITY-MIB::entPhysicalSerialNum.1=STRING:FTX0945W0MY

ENTITY-MIB::entPhysicalSerialNum.2=STRING:

ENTITY-MIB::entPhysicalSerialNum.3=STRING:XXXXXXXXXXX

Itshouldbeimmediatelycleartoyouthatyoucanfindthechassis’sserialnumberbycreatinganitemwith:

ENTITY-MIB::entPhysicalSerialNum["index","ENTITY-MIB::entPhysicalName",

"3745chassis"]

Thenyoucanspecify,inthesameitem,thatitshouldpopulatetheSerialNumberfieldofthehost’sinventory.Thisishowyoucanhaveamoreautomatic,dynamicpopulationofinventoryfields.

Thepossibilitiesareendlessaswe’veonlyjustscratchedthesurfaceofwhatanygivendevicecanexposeasSNMPmetrics.BeforeyougoandfindyourfavoriteOIDstomonitorthough,let’shaveacloserlookattheprecedingexamples,andlet’sdiscussdatatypes.

GettingdatatypesrightWehavealreadyseenhowanOID’svaluehasaspecificdatatypethatisusuallyclearlystatedwiththedefaultsnmpwalkcommand.Intheprecedingexamples,youcanclearlyseethedatatypejustafterthe=sign,beforetheactualvalue.ThereareanumberofSNMPdatatypes—somestillcurrentandsomedeprecated.YoucanfindtheofficiallistanddocumentationinRFC2578(http://tools.ietf.org/html/rfc2578),butlet’shavealookatthemostimportantonesfromtheperspectiveofaZabbixuser:

SNMPtype Description SuggestedZabbixitemtypeandoptions

INTEGERThiscanhavenegativevaluesandisusuallyusedforenumerations

Numericunsigned,decimalStorevalueasisShowwithvaluemappings

STRING Thisisaregularcharacterstringandcancontainnewlines TextStorevalueasis

OID ThisisanSNMPobjectidentifier CharacterStorevalueasis

IpAddress IPv4onlyCharacterStorevalueasis

Counter32 Thisincludesonlynon-negativeandnondecreasingvaluesNumericunsigned,decimalStorevalueasdelta(speedpersecond)

Gauge32 Thisincludesonlynon-negativevalues,whichcandecrease Numericunsigned,decimalStorevalueasis

www.it-ebooks.info

Counter64 Thisincludesnon-negativeandnondecreasing64-bitvalues Numericunsigned,decimalStorevalueasdelta(speedpersecond)

TimeTicks Thisincludesnon-negative,nondecreasingvalues Numericunsigned,decimalStorevalueasis

Firstofall,rememberthattheabovesuggestionsarejustthat—suggestions.Youshouldalwaysevaluatehowtostoreyourdataonacase-by-casebasis,butyou’llprobablyfindthatinmanycasesthoseareindeedthemostusefulsettings.

Movingontotheactualdatatypes,rememberthatthecommandlineSNMPtoolsbydefaultparsethevaluesandshowsomealreadyinterpretedinformation.ThisisespeciallytrueforTimeticksvaluesandforINTEGERvalueswhentheseareusedasenumerations.Inotherwords,youseethefollowingfromthecommandline:

VRRP-MIB::vrrpNotificationCntl.0=INTEGER:disabled(2)

However,whatisactuallypassedasarequestisthebareOID:

1.3.6.1.2.1.68.1.2.0

TheSNMPagentwillrespondwithjustthevalue,which,inthiscase,isthevalue2.

Thismeansthatinthecaseofenumerations,Zabbixwilljustreceiveandstoreanumberandnotthestringdisabled(2)asseenfromthecommandline.Ifyouwanttodisplaymonitoringvaluesthatareabitclearer,youcanapplyvaluemappingstoyournumericitems.Valuemapscontainthemappingbetweennumericvaluesandarbitrarystringrepresentationsforahuman-friendlyrepresentation.Youcanspecifywhichoneyouneedintheitemconfigurationform,asfollows:

www.it-ebooks.info

Zabbixcomeswithafewpredefinedvaluemappings.Youcancreateyourownmappingsbyfollowingtheshowvaluemappingslinkand,providedyouhaveadminrolesonZabbix,you’llbetakentoapagewhereyoucanconfigureallvaluemappingsthatwillbeusedbyZabbix.Fromthere,clickonCreatevaluemapintheupper-rightcornerofthepage,andyou’llbeabletocreateanewmapping.NotallINTEGERvaluesareenumerations,butthosethatareusedassuchwillbeclearlyrecognizablefromyourcommand-linetoolsastheywillbedefinedasINTEGERvaluesbutwillshowastringlabelalongwiththeactualvalue,justasintheprecedingexample.

Ontheotherhand,whentheyarenotusedasenumerations,theycanrepresentdifferentthingsdependingonthecontext.Asseeninthepreviousparagraph,theycanrepresentthenumberofindexesavailableforagivenOID.Theycanalsorepresentapplicationorprotocol-specificvalues,suchasdefaultMTU,defaultTTL,routemetrics,andsoon.

Themaindifferencebetweengauges,counters,andintegersisthatintegerscanassumenegativevalues,whilegaugesandcounterscannot.Inadditiontothat,counterscanonlyincreaseorwraparoundandstartagainfromthebottomoftheirvaluerangeoncetheyreachtheupperlimitsofit.FromtheperspectiveofZabbix,thismarksthedifferenceinhowyou’llwanttostoretheirvalues.

Gaugesareusuallyemployedwhenavaluecanvarywithinagivenrange,suchasthespeedofaninterface,theamountoffreememory,oranylimitsandtimeoutsyoumightfindfornotifications,thenumberofinstances,andsoon.Inallofthesecases,thevaluecanincreaseordecreaseintime,soyou’llwanttostorethemastheyarebecauseonceputonagraph,they’lldrawameaningfulcurve.

Counters,ontheotherhand,canonlyincreasebydefinition.Theyaretypicallyusedtoshowhowmanypacketswereprocessedbyaninterface,howmanyweredropped,howmanyerrorswereencountered,andsoon.Ifyoustorecountervaluesastheyare,you’llfindinyourgraphssomeever-ascendingcurvesthatwon’ttellyouverymuchforyourmonitoringorcapacityplanningpurposes.Thisiswhyyou’llusuallywanttotrackacounter’samountofchangeintime,morethanitsactualvalue.Todothat,Zabbixofferstwodifferentwaystostoredeltasordifferencesbetweensuccessivevalues.

Thedelta(simplechange)storagemethoddoesexactlywhatitsays:itsimplycomputesthedifferencebetweenthecurrentlyreceivedvalueandthepreviouslyreceivedone,andstorestheresult.Itdoesn’ttakeintoconsiderationtheelapsedtimebetweenthetwomeasurements,northefactthattheresultcanevenhaveanegativevalueifthecounteroverflows.Thefactisthatmostofthetime,you’llbeveryinterestedinevaluatinghowmuchtimehaspassedbetweentwodifferentmeasurementsandintreatingcorrectlyanynegativevaluesthatcanappearasaresult.

Thedelta(speedpersecond)willdividethedifferencebetweenthecurrentlyreceivedvalueandthepreviouslyreceivedonebythedifferencebetweenthecurrenttimestampandthepreviousone,asfollows:

(value–prev_value)/(time-prev_time)

www.it-ebooks.info

Thiswillensurethatthescaleofthechangewillalwaysbeconstant,asopposedtothescaleofthesimplechangedelta,whichwillvaryeverytimeyoumodifytheupdateintervaloftheitem,givingyouinconsistentresults.Moreover,thespeed-per-seconddeltawillignoreanynegativevaluesandjustwaitforthenextmeasurement,soyouwon’tfindanyfalsedipsinyourgraphduetooverflowing.

Finally,whileSNMPusesspecificdatatypesforIPaddressesandSNMPOIDs,therearenosuchtypesinZabbix,soyou’llneedtomapthemtosomekindofstringitem.Thesuggestedtypehereischaracterasbothvalueswon’tbebiggerthan255charactersandwon’tcontainanynewlines.

Stringvalues,ontheotherhand,canbequitelongastheSNMPspecificationallowsfor65,535-character-longtexts;however,textthatlongwouldbeoflittlepracticalvalue.Eveniftheyareusuallymuchshorter,stringvaluescanoftencontainnewlinesandbelongerthan255characters.

Consider,forexample,thefollowingSysDescrOIDforthisdevice:

NMPv2-MIB::sysDescr.0=STRING:CiscoIOSSoftware,3700Software(C3745-

ADVENTERPRISEK9_SNA-M),Version12.4(15)T14,RELEASESOFTWARE(fc2)^M

TechnicalSupport:http://www.cisco.com/techsupport^M

CompiledTue17-Aug-1012:56byprod_rel_tea

Asyoucansee,thestringspansmultiplelines,andit’sdefinitelylongerthan255characters.Thisiswhythesuggestedtypeforstringvaluesistextasitallowstextofarbitrarylengthandstructure.Ontheotherhand,ifyou’resurethataspecificOIDvaluewillalwaysbemuchshorterandsimpler,youcancertainlyusethecharacterdatatypeforyourcorrespondingZabbixitem.

Now,youaretrulyreadytogetthemostoutofyourdevices’SNMPagentsasyouarenowabletofindtheOIDyouwanttomonitorandmapthemperfectlytoZabbixitems,downtohowtostorethevalues,theirdatatypes,withwhatfrequency,andwithanyvaluemappingthatmightbenecessary.

It’snowtimetoexploretheotheraspectofSNMP:traps.

www.it-ebooks.info

SNMPtrapsSNMPtrapsareabitofanoddballwhencomparedtoalltheotherZabbixitemtypes.Unlikeotheritems,SNMPtrapsdonotreportasimplemeasurement,butaneventofsometype.Inotherwords,theyaretheresultofsomekindofcheckorcomputationmadebytheSNMPagentandsentovertothemonitoringserverasastatusreport.AnSNMPtrapcanbeissuedeverytimeahostisrebooted,aninterfaceisdown,adiskisdamaged,oraUPShaslostpowerandiskeepingserversupusingitsbattery.

ThiskindofinformationcontrastswithZabbix’sbasicassumptionthatanitemisasimplemetricnotdirectlyrelatedtoaspecificevent.Ontheotherhand,there’snootherwaytobeawareofcertainsituationsifnotthroughanSNMPtrapeitherbecausetherearenorelatedmetrics(consider,forexample,theeventtheserverisbeingshutdown)orbecausetheappliance’sonlywaytoconveyitsstatusisthroughabunchofSNMPobjectsandtraps.

SotrapsareofrelativelylimitedusetoZabbixasyoucan’tdomuchmorethanbuildasimpletriggeroutofeverytrapandthennotifyabouttheevent(notmuchpointingraphingatraporbuildingcalculateditemsonit).Nevertheless,theymightproveessentialforacompletemonitoringsolution.

TomanageSNMPtrapseffectively,Zabbixneedsacoupleofhelpertools:thesnmptrapddaemontoactuallyhandleconnectionsfromtheSNMPagentsandsomekindofscripttocorrectlyformateverytrapandpassittotheZabbixserverforfurtherprocessing.

SnmptrapdIfyouhavecompiledSNMPsupportintotheZabbixserver,youshouldalreadyhavethecompleteSNMPsuiteinstalled,whichcontainstheSNMPdaemonandtheSNMPtrapdaemonalongwiththeutilitieswehaveusedintheprevioussection.

JustastheZabbixserverhasabunchofdaemonprocessesthatlistenonTCPport10051forincomingconnections(fromagents,proxies,andnodes),snmptrapdisthedaemonprocessthatlistensonUDPport162forincomingtrapscomingfromremoteSNMPagents.

Onceinstalled,snmptrapdreadsitsconfigurationoptionsfromansnmptrapd.conffilethatcanbeusuallyfoundinthe/etc/snmp/directory.ThebareminimumconfigurationforsnmptrapdrequiresthedefinitionofauserandaprivacylevelforSNMPv3,asfollows:

createUserzbxuserSHAauthAESpriv

authUserlog,execute,netzbxuser

TipTheaboveconfigurationwillenablesnmptrapdtoreceiveSNMPv3INFORMpackets.ThesearejustlikeregularSNMPtraps,withtwodifferences:thefirstoneisthatwhileanagentwon’texpectaresponseaftersendingatrap,INFORMpacketsareacknowledged,sosnmptrapdwillsendaresponseforeverytrapreceived.Butthemostimportantdifference

www.it-ebooks.info

isthatwithINFORMpackets,theauthoritativeEngineIDwillbethatofthereceivingpartyandnotthesendingpartyaswithregulartraps.Thismeansthatyou’llhavetospecifyyourserver’sEngineIDtoeverydevicethatwillsendSNMPv3INFORMpackets.Sinceyou’llhavetoconfigurethemtosendpacketstotheserveranyway,thiswon’tmeantoomuchwork.Manyagentsautomaticallydiscoverapeer’sEngineIDbeforesendinganINFORM,butifyouneedtosetityourself,youcandiscoveryourserver’sEngineIDusingsnmpgetandaskingforthesnmpEngineID.0OID.

IfyouwanttouseregularSNMPtraps,you’llhavetoinsertanewcreateUserlineforeveryagentthatwillsendtrapstotheserver,witheachonespecifyingthecorrectEngineIDoftheagentsendingtraps.

Withthisminimalconfiguration,snmptrapdwilllimititselftologthetraptosyslog.WhileitcouldbepossibletoextractthisinformationandsendittoZabbix,it’seasiertotellsnmptrapdhowitshouldhandletraps.Whilethedaemonhasnoprocessingcapabilitiesofitsown,itcanexecuteanycommandorapplicationeitherusingthetrapHandledirective,orleveragingitsembeddedPerlfunctionality.Thelatterismoreefficientasthedaemonwon’thavetoforkanewprocessandwaitforitsexecutiontofinish,soit’stherecommendedoneifyouplantoreceiveasignificantnumberoftraps.Justaddthefollowinglinetosnmptrapd.conf:

perldo"/usr/local/bin/zabbix_trap_receiver.pl";

TipYoucangetthezabbix_trap_receiverscriptfromtheZabbixsources.It’slocatedinmisc/snmptrap/zabbix_trap_receiver.pl.

BesuretocheckthatyoualsohavetheNet-SNMPPerlmoduleinstalled.Ifyouneedit,asimpleyuminstallnet-snmp-perlcommandshouldtakecareofeverything.

Oncerestarted,thesnmptrapddaemonwillexecutethePerlscriptyouspecifiedtoprocesseverytrapreceived,translatingitintoaformatthatcanbeeasilyparsedbytheZabbixserver.Inthefollowingsection,we’llseehowanSNMPtrapistranslatedandusedbyZabbix.

TransformingatrapintoaZabbixitemThePerlscriptincludedintheZabbixdistributionworksasatranslatorfromanSNMPtrapformattoaZabbixitemmeasurement.Foreverytrapreceived,itwillformatitaccordingtotherulesdefinedinthescriptandwilloutputtheresultinalogfile.Bydefault,thelogfileiscalled/tmp/zabbix_traps.tmp.YouneedtomakesurethatthesamefileisreadbyZabbixbysettingthefollowingparametersin/etc/zabbix/zabbix_server.conf:

###Option:StartSNMPTrapper

#If1,SNMPtrapperprocessisstarted.

#Mandatory:no

#Range:0-1

#Default:

www.it-ebooks.info

StartSNMPTrapper=1

###Option:SNMPTrapperFile

#TemporaryfileusedforpassingdatafromSNMPtrapdaemontothe

server.

#Mustbethesameasinzabbix_trap_receiver.plorSNMPTT

configurationfile.

SNMPTrapperFile=/tmp/zabbix_traps.tmp

Thelogfilewillhaveaformatsimilartothefollowingexample:

03:47:102014/12/09ZBXTRAP127.0.0.1

PDUINFO:

notificationtypeTRAP

version0

receivedfromUDP:[127.0.0.1]:34373->[127.0.0.1]

errorstatus0

messageid0

communitypublic

transactionid3

errorindex0

requestid0

VARBINDS:

DISMAN-EVENT-MIB::sysUpTimeInstancetype=67value=Timeticks:(55)

0:00:00.55

SNMPv2-MIB::snmpTrapOID.0type=6value=OID:IF-MIB::linkDown.0.33

IF-MIB::linkDowntype=4value=Hex-STRING:E2809C5445

53544D454E4F57E2809D

SNMP-COMMUNITY-MIB::snmpTrapCommunity.0type=4value=STRING:"public"

SNMPv2-MIB::snmpTrapEnterprise.0type=6value=OID:IF-MIB::linkDown

TheZBXTRAPfollowedbytheIPaddresswillmarkthestartofanewlogstanza.Therestofthelogwillcontainalldetailsaboutthetrap,soyou’llbeabletoactonanyofthose.

TheZabbixserverwillinturnmonitortheaforesaidlogfileandprocesseverynewlineasanSNMPtrapitem,basicallymatchingthecontentofthelogtoanytrapitemdefinedfortherelevanthost.

Asyou’vealreadyseen,thefirstpartoftheloglineisusedbytheZabbixtrapreceivertomatchatrapwithitscorrespondinghost.Therestismatchedtotheaforesaidhost’sSNMPtrapitem’sregexpdefinitionsanditscontentaddedtoeverymatchingitem’shistoryofvalues.ThismeansthatifyouwishtohavealinkDowntrapitemforagivenhost,you’llneedtoconfigureanSNMPtrapitemwithansnmptrap["linkDown"]key,asfollows:

www.it-ebooks.info

Youmightneedtomakesurethatthelogtimeformatyouspecifyintheitem’sconfigurationwillmatchtheoneusedbythePerlscript.You’llalsohavetocheckthatthehost’sinterfacewillmatchtheoneloggedbysnmptrapdbecauseit’stheonepieceofdataZabbixwillusetomatchtrapstohosts.

Fromnowon,you’llbeabletoseethecontentsofthetrapintheitem’sdatahistory.

MovingonfromSNMP,therearestillotherdatasourcesthatyoucanrelyontogetmonitoringdataintoZabbix;forthepurposesofthisbook,themostinterestingonesarelogfiles.ComparedtoSNMP,theycanbetrickytoworkwith,buttheydohavetheiruses,solet’sexplorethemforawhile.

GettingnetflowfromthedevicestothemonitoringserverNetflowisaprotocoloriginallydevelopedbyCiscotocollectandmonitorstatisticsofnetworktrafficonadevice.Aftertheinitialrelease,manyvendorsstartedprovidingtheirownimplementationoftheprotocol.In2008IETFstandardizednetflowandpublishedInternetProtocolFlowInformationeXport(IPFIX)basedonnetflowv9withsomeextensions.However,netflowsomehowremainstheexistingnameoftheprotocolinfactbutnotnecessarilybylegalright,sothat’stheonewe’llusehere.

Anetflowrecordcontainsinformationaboutasinglenetworkflow.Aflowisasequenceofpacketsthatsharesomecommonproperties:

IPprotocolSourceIPaddressSourceport(forTCPandUDP)DestinationIPaddressDestinationport(forTCPandUDP)InputinterfaceTypeofservice

Foreachflow,arecordexposesmanydifferentvalues,whichchangewithnetflow

www.it-ebooks.info

versionsandimplementations.Herearethemostcommonones:

InputinterfaceofthedeviceOutputinterfaceofthedeviceFlowstarttimeFlowendtimeNumberofbytesintheflowNumberofpacketsintheflowSourceIPaddressSourceIPportSourceIPmaskDestinationIPaddressDestinationIPportDestinationIPmaskICMPtypeandcodeTCPflagsIPaddressoftheimmediatenext-hop

Itshouldbeimmediatelycleartoyouthatthistypeofinformationcanbeextremelyusefultoanetworkadministratorasitallowsyoutobuildapictureofallthetraffictraversingyournetwork.ItcanalsobeusedtoidentifyanomaloustrafficandtraffictoandfromIPaddressesorportsthatshouldnotbethere,orasforensicevidenceafteranincident.Moreover,itcanbeusedasasourceforcapacity-planninganalysistoidentifybottlenecksinyournetwork,periodsofpeakuse,andtoptalkersamongyourserversanddevices.

Finally,aswewereexplainingpreviously,it’sagoodcandidateforaZabbixlogitemasflowdataisusefulevenifitisnotdirectlyrelatedtothehostthatgeneratedit(evenifit’sstillusefultotrackthatpieceofinformationwheneverpossible).

So,let’sseehowtogetnetflowdataintoZabbix.

Firstofall,you’llhavetoconfigureyourdevicetosendflowdatatoaserver.InthecaseofaCiscodevice,herearetheconfigurationcommandsthatyouneedtoissue(remembertosubstituteallreferencestotheexampleZabbixserverwiththerealonesthatapplytoyourenvironment):

R1(config)#ipflow-exportdestination192.168.234.1319995

R1(config)#ipflow-exportversion9

R1(config)#interfacef0/0

R1(config-if)#ipflowingress

R1(config-if)#ipflowegress

R1(config-if)#exit

Inthefirstline,wespecifytheIPaddressofourZabbixserverandtheUDPportthedeviceshouldsendnetflowinformationto.

Thesecondlinesetsthenetflowversion.

Inthethirdline,wegointointerfacef0/0mode.Pleasenotethatyou’llhavetoexplicitlyenablenetflowforeveryinterfaceyouareinterestedin.Thisisusuallynotaproblem

www.it-ebooks.info

becauseifyouconfigurenetflowontherightinterfacesofyourrouters,you’llseemost,ifnotallofyourtrafficanyway;youwon’tneedtoenablenetflowoneveryinterfaceofeverynetworkdeviceyouhave.

Thefourthlineenablesnetflowmonitoringforincomingtrafficoninterfacef0/0,whilethefifthlineenablesnetflowmonitoringforoutgoingtrafficonthesameinterface.Ifyouwanttoenablenetflowonotherinterfaces,you’llneedtorepeatlines3to5foreveryinterfaceyouareinterestedin.

Repeatthewholeprocessforalltheroutersyouwanttogetflowinformationfrom,andonceyouaredone,youarereadytoturntoyourZabbixserver.

ReceivingnetflowdataonyourserverToactuallyreceiveandprocessnetflowpacketsonaserver,youneedadaemonthatwilllistenonaspecifiedUDPport,andthatwillunderstandthenetflowprotocol.OnLinux,suchdaemonsandassociatedtoolsarecontainedinthenfdumppackage.

Nfdumpisacollectionoftoolsthatwillenableyoutocapturenetflowdata,storeitondisk,filterit,andanalyzeit.Themostimportantcomponentsare:

nfcapd:Thisisthedaemoncomponentthatlistensforincomingnetflowdataandstoresitondiskinbinaryformatnfdump:Thisissimilartotcpdump;itreadsandfiltersnfcapdfiles,andoutputsreadabledata

Sothebasicdataflowwillbesimilartothisone:

1. Aroutersendsnetflowdatatotheserver.2. Ontheserver,nfcapdcapturesthedataandstoresitinbinaryfiles.3. Aschedulednfdumpprocesswillreadthebinaryfilesandpopulateahumanreadable

logwithnetflowinformation.4. AZabbixagentwillreadthelogandsenddatatotheZabbixserveraccordingtothe

item’sconfiguration.

Wehavealreadytakencareofpoint1,solet’sseehowtoinstallandconfigurethenfdumppackage,beforelookingintotheZabbixside.

Unfortunately,therearenoreadymaderpmpacketsfornfdump,sowe’llneedtofindthesourcecode,compileit,andinstallit.Thisisusuallyastraightforwardprocess.Firstofall,let’sinstallsomerequireddependenciesfornfdump:

#yuminstallrrdtoolrrdtool-develrrdtool-docperl-rrdtool

Then,we’llneedtodownloadthelatestsources.Atthemomentofwritingthis,thelatestavailableversionis1.6.12.Youcandownloadthepackagefromhttp://sourceforge.net/projects/nfdump/andthentransferittoyourserver.Onceyouhavetar.gzready,unpackit:

$tarxvzfnfdump-1.6.12.tar.gz

www.it-ebooks.info

Thenmoveintothenfdump-1.6.12directoryandruntheusualconfigure,make,andmakeinstallsequence.Ifyouwanttoinstallnfdumpinthemaindirectoriesinsteadofthe/usr/localtree,justpassthe–prefixoptiontotheconfigurescript.Inthefollowingexample,that’swhatwe’lluse:

$cdnfdump-1.6.12

$./configure–prefix=/usr--sysconfdir=/etc

$suroot

#makeinstall

Onceinstalled,youcanaddadedicateduserfornfcapdsothatitdoesn’thavetorunasrootandsetaworkingdirectoryforit:

#useradd-s/sbin/nologinnetflow

#mkdir-p/var/nfdump/nfcapd

#mkdir-p/var/nfdump/logs

#chown-Rnetflow/var/nfdump

Whenyourunnfcapd,itwillcreateitsbinaryfilesunder/var/nfdump/nfcapd.Nfcapdfilesarerotated,bydefault,onceeveryfiveminutesandcanbeseparatedintoonedumpcollection(currentandrotatedfiles)persendinghostorasinglecollectionforallsendinghosts.Theycanalsobeexpiredafterasetamountoftime.Youarenowreadytowaitfornetflowdataandtransformitintoalogfile.Todothat,you’llneedtopasstherightoptiontonfcapd.Sincetherearequiteafewoptionstopass,let’sbuildthecommandlinelittlebylittle.Pleasedon’truntheintermediatecommands,butonlythefinalone;nfcapdwillcomplainaboutmissingoptionsandrefusetorun.

Firstofall,we’llpasssomeoptionsthatwillinstructnfcapdtogointodaemonmode(-D),tocompressoutput(-z),torunasusernetflow(-u),andtolistenonport9995(-p):

#nfcapd-D-z-unetflow-p9995

Then,we’llneedtoaddsomeoptionsaboutdatasources.Theacceptedcurrentmethodistousethe-nswitch.We’llalsoinstructnfcapdtocreateadditionalsubdirectoriestostorethecapfilestobetterorganizethem(-S):

#nfcapd-D-z-unetflow-p9995-nR1,192.168.11.9,/var/nfdump/nfcapd-n

R2,10.10.1.254,/var/nfdump/nfcapd-S2

Asyoucansee,you’llhavetospecifyadifferent-noptionforeverysourceyouconfigure.Ifyouhavemanynetflowsources,itmightbebettertorundifferentinstancesofnfcapdondifferentUDPportssoastosharetheloadbetweendifferentprocesses.Inthatcase,justremembertoconfigureyourdevicesaccordinglysothattheysendtheirtraffictothecorrectUDPport.The-S2optionwillcreateadditionalyear/month/day/hourdirectoriesunder/var/nfdump/nfcapdtostorecurrentandrotatedfiles.

Nfcapdfilesarerotatedeveryfiveminutes,andifyournetworkhasalotoftraffic,yournfcapddirectorycanbecomehuge.Youcouldscheduleaseparatejobtocleanthemup,butwiththe-eoption,nfcapdwillbeabletoalsotakecareofthat.Justsettheexpirationparameterwithnfexpireandnfcapdwillpickthemup:

www.it-ebooks.info

#nfexpire-u/var/nfdump/nfcapd-s15G-t90d

R2,10.10.1.254,/var/nfdump/nfcapd-S2-e

Intheaboveexample,wesetthesizelimitofthedirectoryto15gigabytes,andthecap(maximum)fileageto90days.Fileswillbedeletedbynfcapdwheneveroneoftheselimitsisreached.Thelastlineintheprecedingcommandnowcontainsalltheparametersweneedforbasicnetflowdumping.Ifyourunit(don’tforgetthenfexpirecommandtoo)orputitintoastartupscript,nfcapdwilllistenonthespecifiednetworkportforincomingnetflowdataandwriteittothedirectoriesyouspecified.

Onceyouhavesomedatain,youcanreaditwithnfdumpandoutputahuman-readablesetofrecords:

$nfdump-r/var/nfdump/nfcapd/2014/10/29/02/nfcapd.201410290250-o

extended

DateflowstartDurationProtoSrcIPAddr:PortDstIP

Addr:PortFlagsTosPacketsBytesppsbpsBppFlows

2014-10-2902:51:53.16063.545TCP10.13.27.151:80->

123.43.98.124:6523.AP.SF01288412055056

2014-10-2902:53:13.37023.135TCP64.76.73.121:25->

10.138.41.151:7643.AP.SF0512450055156

Timewindow:Oct29201402:50:00-Oct29201402:54:56

Thisisgettingclosertoourobjective.Ifyourunnfdumpandredirectitsoutputtoafileinsteadofthescreen,thereyouhavethelogfilewe’vebeentalkingaboutinthelastseveralpages.Todothat,youareprobablythinkingofsettingupacronjobthatwillfindthelatestnfcapdfilesthatweren’talreadyparsedbynfdump,makenfdumpreadthemwhilespecifyingatimewindowsothatyourlogfilewon’tcontainduplicateddata,andaddtheaforesaidoutputtoalogfilethatwillbemonitoredbyZabbix.Thiscanbeanontrivialexercisewhenyouconsiderthatnfcapdwillcontinuallyproducenewfilesandwillputtheminnewdirectoriesallthetime.Moreover,you’llneedtokeepsomekindofexecutionstatewiththetimestampofthelasttimenfdumpwasruninordertoavoidtheaforesaidduplicates.

Itturnsoutthatyou’llbeabletoavoidallthiswork,thankstoaniceoptionfornfcapd,the-xoption.Solet’srewritethenfcapdcommandonelasttime:

R2,10.10.1.254,/var/nfdump/nfcapd-S2-e-x'nfdump-q-oextended-r%d/%f

>>/var/nfdump/logs/zabbix_netflow.log'

The-xcommandexecutesanarbitrarycommandeverytimeadumpfileisrotated.Youcanreferencethedumpfileandthebasedirectorywiththe%d/%fmacros.Thismeansthatnfdumpwillalwaysbeexecutedonnewdataandonlyonceperdumpfile.Suddenly,youwon’tneedtoscheduleanycomplicatedcronjobtogeneratethefinal,human-readablenetflowlogfile.Wealsoaddeda-qoptiontosuppresstheheaderandstatisticsprintingtokeepthelogfileclean.

www.it-ebooks.info

NoteYoumightstillwanttoconfiguresomelogrotationforthe/var/nfdump/logs/zabbix_netflow.logfile.Ifyouletitgrowunchecked,itwillfillupyourdiskspaceinduetime!

It’sfinallytimetomakeZabbixawareofthenetflowlogfile.

MonitoringalogfilewithZabbixAsalreadyexplained,logfilemonitoringneedsaZabbixagent.Forillustrationpurposes,wewillassumethatyouhaveinstallednfdumponthesameboxastheZabbixserver,andthatthelogfileisthuslocallyavailable.Itgoeswithoutsayingthatyoucouldalsoinstallnfdump,alongwithaZabbixagent,onaseparated,possiblydedicatedmachine.Itwon’tmakeanydifferencefromZabbix’sperspective.

Basicitemcreationisfairlystraightforward,justpointtheitemkeytothecorrectfilepathandyou’regoodtogo.Pleasenote,inthefollowingexample,thetimestampparsingfield:

Thisisallyouneedforbasiclogfilemonitoring.Forfurtherexplorations,thelogkeyacceptsdifferentoptions,amongwhichthemostinterestingarethoserelatedtoregularexpressionfilteringandoutputsothatyoucanalsocreateadditionalitemsthatwillonlyextracttheexactinformationyouneed(forinstance,bytespersecondofaflow)anduseitasrawdata,justasyouwoulduseanyotherZabbixitem.Zabbix’sownofficialdocumentationisexcellentinthisrespect,soyouareencouragedtofindoutmoreathttps://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/log_items.

www.it-ebooks.info

Onthenfdumpside,therearemanymoreoptionsandfeaturesavailabletonfdump,we’vereallyonlyscratchedthesurfacetokeepthingssimple.Wedon’thavethespacetofullyexploreithere,butifyou’rewillingtospendsometimeexploringthetool,you’llfindthatnfdumpisnotonlycapableofpowerfultrafficfiltering,justastcpdumpis,butitcanalsocreatestatisticsandaggregateddataonvirtuallyeveryaspectofaflow,fromnetworkportstopacketsizes,andsoon.CombinethiswithZabbix’spowerfulexternalscriptitems,andyoucaneasilyseethatyoucansliceanddiceyourdata;however,ifyouwant,bringitintoZabbixforfurtherprocessing,graphing,andalarming.Really,theskyisthelimitwhenyoulearntocombinethesetoolstogether.

www.it-ebooks.info

SummaryInthischapter,youhavelearnedthedifferentpossibilitiesZabbixofferstotheenterprisingnetworkadministrator.

Youshouldnowbeabletochoose,design,andimplementallthemonitoringitemsyouneed,basedonthemethodsillustratedintheprecedingparagraphs:simplechecksthataremoreusefulandpowerfulthanthenameimplies;theall-powerfulSNMPprotocol,bothasgetvaluesandastraps;logfilesingeneral;andtheinfinitelyusefulnetflowprotocol

ThenextchapterwillbuildontheinformationexposedinthischapterandwillfocusmoreonservermonitoringandhowtoextractinformationfromDNSservers,webservers,proxies,andotherappliances.Theseareimportant,ifoftenoverlooked,componentsofanetworkevenfromtheperspectiveofanetworkadministrator,andyou’llfindmanyusefultipsonhowtomonitorthem.

www.it-ebooks.info

Chapter3.MonitoringYourNetworkServicesIneveryenvironment,especiallyinalargeone,therearemanynetworkcriticalservicesthataredirectlytiedonthenetworkinfrastructure.Manyofthemcanbemonitoredbythesystemadministrators,butthecorecriticalservicesforthewholenetworkarebetteriftheyaremonitoreddirectlybythenetworkadministrator.

Betweenthosecriticalservices,wecanfindthefollowing:

DNSDHCPNTPApacheproxy/reverseproxiesProxycacheSquid

Asitiseasytounderstand,evenifthoseservicesareprovidedfromsomededicatedserverandnotnetworkdevices,themetricsthatyouareacquiringfromthemarefundamental.Thosemetrics,indeed,playacriticalrolewhenyouwouldliketosetupaproactivealarm.

AnexampleofaservicethatcancausealotofconfusioninyournetworkcanbetheDNS,theDHCP,oreventheNTP.Inanidealenvironment,allthoseservicesneedtoberesponsive,andeventheresponsetimeiscrucial;ifeachoneofthosecomponentsbecomesunresponsive,theywillactastheweakestlinkofyourinfrastructure,causingalotofproblemsthatwillbequicklypropagatedtothewholenetwork.AsimpleNTPservercanintroduceconfusioninthelogsofyoursystemsorevencauseanissueinyourconnections.Workingonapracticalexample,trytoimaginethatyouhaveallyouraccountsstoredinanLDAP.Well,iftheLDAPtakestoomuchtimetoresolvetheUID/GIDofyouraccount,youcanhaveissuespropagatedtoallyoursystems.AnunresponsiveLDAPcancausefilesystemissuesandevenNASissues,andifallyouraccountsarestoredthere,evenanlscanliterallytakeages,withabigimpactonthewholeinfrastructure.Here,wearenotconsideringtheDNS,whereadysfunctioncanbeevenworse.

Also,thoseservicesneedtobetakenundersurveillanceas,iftheybecomeunresponsive,quitesoontheywillaccumulaterequeststoserve,andiftheenvironmentisnotready,theywillbefloodedbytheirownqueriesinaqueue,withaglobalimpactonourinfrastructure.

Inthischapter,wewillgothroughallthemainservicesthatanetworkadminshouldmonitortoavoidthesekindsofissues.Then,thereaderwilllearnandunderstandtheimportanceofaneffectiveproactivealarmtoavoidaquickescalationofissuesacrossthenetwork.

www.it-ebooks.info

MonitoringtheDNSThefirstnetworkcomponentwewillanalyzeandseehowtomonitoristheDNS.

ThemostpopularDNSserverisBIND,whichisalsooneoftheoldestpackagesproduced.Here,inthenextexample,weassumeyouhaveBIND9.6orlater.

Startingwithversion9.6,thereisabrandnewfeaturethatisnotevenmentionedinthemainpage(ofRedHatLinuxatleast).Thisfeatureisabuilt-inwebserverthatprovidesstatisticsaboutBINDinaverysimplewaythoughtHTTP.Toenablethisfeature,itisenoughtoaddthoselinestoyourBIND9configurationfile,/etc/named.conf:

statistics-channels{

inet127.0.0.1port8053allow{127.0.0.1;};

Thelinewehavejustaddedisagoodexampleasthestatistics’accessiscontrolledandrestrictedtothelocalhost.

TipBIND,bydefault,willusethestandard80HTTPportifyoudon’tspecifytheport.Alsopleasetakecaretolimittheaccesstothestatisticchannel;todoso,youcanusethisclause:

allow{address_match_list}

Ifyoudon’tspecifytheallowclause,BINDwillacceptconnectionsfromanyaddress.Thisneedstobeavoided.

Oncethisisdone,allyouhavetodoisrestartyourservicewith:

$servicenamedrestart

Stoppingnamed:[OK]

Startingnamed:[OK]

Now,youcanevenusecurltocallyourwebserverandhavedeliveredtoyouallthestatistics:

#curlhttp://127.0.0.1:8053

<?xmlversion="1.0"encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl"href="/bind9.xsl"?>

<iscversion="1.0">

<bind>

<statisticsversion="2.2">

<views>

<view>

<name>_default</name>

<zones>

www.it-ebooks.info

</summary>

</memory>

</statistics>

</bind>

</isc>

Now,wehavetwowaystoretrievethestatistics:

ConfigureBINDtowritethestatisticsinthestatfile(oldmethod)ConfigureBINDtousethebuilt-inHTTPwebservice

Thefirstandoldmethodcanbeusedforserversthatarenotunderaheavyload;thenewmethodusingthestatistics-channelsisontheotherhandlightweightandveryeasytomanage.Nowadaysthisoneisthepreferredmethodtouse.

NoteStartingfromBIND9.10,thestatisticscanbedeliveredineithertheXMLortheJSONformat.ThepreviousversionofBINDofferedonlystatisticsonXMLv2orV3.StartingwithBIND9.10,theXMLstatisticsareavailableonlyinV3format.Anyway,theJSONformatissignificantlyfasterthanXMLandevenlightweighttoprovide.

Now,tofiltertheoutputobtainedbycurl,thereisaninterestingutilitythatunfortunatelyisnotastandardRPMdistributedbyRedHat.Thetoolwearegoingtouseonthoseexamplesisxml2.

Thisxml2isanXMLprocessingtoolthatcanbeusedtoparseandreadtheXMLenvelopesandrewritethemasaflatformat.Theflatformatisreallyusefultobemanipulatedwithshellscripts.Then,firstofall,youneedtodownloadthisutility(thesourcecodeisavailableathttp://download.ofb.net/gale/xml2-0.5.tar.gz).Here’stheoutputsummary:

#wgethttp://download.ofb.net/gale/xml2-0.5.tar.gz

--2014-11-0110:43:44--http://download.ofb.net/gale/xml2-0.5.tar.gz

Resolvingdownload.ofb.net…64.13.131.34

Connectingtodownload.ofb.net|64.13.131.34|:80…connected.

HTTPrequestsent,awaitingresponse…200OK

Length:86318(84K)[application/x-gzip]

Savingto:"xml2-0.5.tar.gz"

100%[===================================>]86,318155K/sin0.5s

2014-11-0110:43:45(155KB/s)-"xml2-0.5.tar.gz"saved[86318/86318]

Performthefollowingstepstoobtaintheresultssetoutintheprecedingparagraph:

1. Explodethepackage,asfollows:

#tar-zxvfxml2-0.5.tar.gz

xml2-0.5/

xml2-0.5/configure.ac

xml2-0.5/aclocal.m4

www.it-ebooks.info

xml2-0.5/csv2.c

xml2-0.5/xml2.c

2. Stepintothedirectory,asfollows:

#cdxml2-0.5

3. Runtheusual./configurefollowedbymakeandmakeinstall,asfollows:

#./configure&&make

Then,asroot,youcannowrunthefollowingcommand:

#makeinstall

Onceallthishasbeencompleted,youarereadytoruntheutility.

Tomakeyoubetterunderstandwhatthistoolexactlydoes,youcanrunthefollowingcommand:

#curlhttp://localhost:8053/2>/dev/null|xml2|grep-A1queries

/isc/bind/statistics/server/queries-in/rdtype/name=A

/isc/bind/statistics/server/queries-in/rdtype/counter=11230

/isc/bind/statistics/server/queries-in/rdtype

/isc/bind/statistics/server/queries-in/rdtype/name=AAAA

/isc/bind/statistics/server/queries-in/rdtype/counter=1112

Now,theoutputisfinallyveryeasytomanipulatewithastandardutilitylikesedorawk.

4. Then,thenextsteptoenquirefromthelocallyinstalledagentistoaddthesetwolines:

UserParameter=bind.queries.in[*],curlhttp://localhost:8053/

2>/dev/null|/usr/local/bin/xml2|grep-A1

"/isc/bind/statistics/server/queries-in/rdtype/name=$1$"|tail-1|

cut-d=-f2

UserParameter=bind.queries.out[*],curlhttp://localhost:8053/

2>/dev/null|/usr/local/bin/xml2|grep-A1

"/isc/bind/statistics/views/view/rdtype/name=$1$"|tail-1|cut-d=-

Usingtheprecedingcommandasanexample,youcanrunthestandardqueries,suchasA,AAAA,CNAME,ANY,MX,NS,PTR,SOA,andTXTrecordsin/out.

Now,ontheZabbixserverside,youneedtoconfigureallyouritemsjustastheoneshowninthescreenshotfollowingtheupcominglist,takingcaretocreatethesamekindofitemforAaswell:

www.it-ebooks.info

Onceyou’veaddedallyouritemsinagraph,thefinalresultwillbejustliketheoneshowninthenextscreenshot.Now,you’reacquiringallthequeriesdoneforthemostimportantDNSfields.

www.it-ebooks.info

DNS–responsetimeNow,wearemonitoringallqueriesdoneagainstallthemainDNSrecords,butactuallyweneedtocheckhowourDNSisworkingandthenhowmuchtimeisrequiredtohavetheresponseback.

OntheZabbixhow-to,thereisanexampletodowhat’savailablehere:https://www.zabbix.com/wiki/doku.php?id=howto/monitor/services/monitor_dns_and_ntp_services_on_your_network.

Theproblemwiththisexampleisthatthescriptandcodeproposedsimplyreturnsa0or1dependingontheDNSresponseorDNStimeout.

Well,thatexampleisnotgoodenoughforus;wearelookingfornumberslikeresponsetime,andoverthosenumberswecanimplementatrigger.ThetriggerneedstogoonfirewhenthetimeneededbyDNStogiveusbackaresponseishigherthanavaluethatwecanconsideracceptable.Inacomplexnetwork,youcanhaveaDNSquerywhereyoucantolerateaslowresponse(theentiredevelopmentnetworksegment,forinstance,isnotascriticalastheproductionsegment).Then,thesolutionsweproposeheregiveustheresponsetime.Wecanbuildourtriggerovertheresponsetimeunliketheotherway,whichisalotlessflexible.

Wecanseethescriptstepbystep;firstofall,weneedtoacquiretheresponsetime.Thiscanbedoneusingdig,asfollows:

#digmydomain.com

NoteNOTE

digispartofthebind-utilspackage.Ifyoudon’thaveitinstalledinyoursystem,youneedtorunasrootthefollowingcommand:

yuminstallbind-utils

Anyway,digusesthelocalresolver,andthenifyourunthesamequeryagain,you’llseethatthetimespenttoacquiretheDNSrecordis0minutes.Thisisclearlyafalsevalue!Toavoidanycachedresponseandtomeasuretherealtime,weneedtousethe+traceoption.Whentracingisenabled,digmakesiterativequeriestoresolvethename;practically,digwillfollowreferralsfromtherootservers,showingtheanswerfromeachserverthatwasusedtoresolvethelookup.

Here,weneedtohavethetotaltimespentforthequeryandnotthetimeconsumedbyeveryserver.Todothat,wecanusethefollowingsyntax:

$(timedig@127.0.0.1mydomain.com+trace)

real0m1.376s

user0m0.010s

sys0m0.012s

Nowthatwehaveunderstoodthelogic,hereisthefullscriptwewilluse:

www.it-ebooks.info

#cattest_dns.sh

#!/bin/sh

iftest-z"$1";then

echo"YouneedtosupplyaDNSentrytocheck.Quitting"

exit01;

DOMAIN=$1

MYTIME=$((timedig$DOMAIN+trace)2>&1|grepreal|awk-F'[m,s]''{print

if[$?-eq0];then

echo$MYTIME

Thisscriptrequiresa$1parameter,whichisthedomaintocheck.Now,weneedtoenablethisscriptontheagent’ssidewithUserParameterontheagentconfigurationfile,thusadding:

UserParameter=dns.responsetime[*],test_dns.sh$1

Thescriptwejustcreatedneedstoplacedinavalidruntimeagent’spath,orweneedtousethefullyqualifiedpathinUserParameter,asfollows:

UserParameter=dns.responsetime[*],/full/path/of/test_dns.sh$1

NoteThismethodisreallyusefulasyoucandeploythescriptondifferentnetworksegments,likeforinstance,theapplicationserverzone,andhavearealvalueofthetimeneededtoresolveaDNShostfromthatnetworksegment.

Asthelaststep,createtherelativeitemontheZabbixserverside,whereyouwillpasstheDNSnametocheck,asshowninthefollowingscreenshot:

Pleasebearinmindthatthisscript,ifexecutedcontinuously,canhammeryourDNSexactlybecauseitavoidsusingthecacheofthelocalresolverandevenoneoftheintermediatesegments.

Then,aswehaveexplained,weneedtoscheduleourscriptwithareasonableperiodthatcanbeforaninstanceof1minute.Pleaseconsideryournetworksegmentsfromwhichyou’rerunningthischeck,forboththequantityofscriptsthatarerunningandfrequency.

www.it-ebooks.info

NoteHere,youcancreateatriggerbasedonthezone,bearinginmindthatyou’remonitoringtheDNSresponsetimedirectlyfromthehoststhatrequirethoseDNSentriesresolved.Here,itisimportanttotuneyourtriggerbasedontheresponsetimeyouconsideracceptablefromthepointofviewofthezone.

Whenyou’recreatingyourtrigger,itisimportanttoconsiderthatthispluginprovidesyouwiththerealDNSresponsetime,whichistheworst-casescenario.Here,weavoidusinganycachingsystems,whichisnottherealcasebutapessimisticone.Thatsaid,ifyounoticesomespikesofhighresponsetime,thosecanbeignoredasthosespikescan’timpactyoursystem.Consideringthat,thetriggerneedstobetunedtospottheresponsetimethatisstilltherefortwoorthreeitemcycles(orevenmore—thisdependsonthefrequencyatwhichyourunthecheck)andavoidconsideringsinglespikes.

www.it-ebooks.info

DNSSEC–monitoringthezonerolloverHere,wedon’thaveenoughpagestoexplainallthefeaturesaddedbyDNSSECoracompletesetupguideofit.Anyway,itisimportanttoknowthatthebestwaytoavoidissueslikeaDNScachepoisoningattackistouseDNSSEC.DNSSECdoesadeepusageofcryptographickeysanddigitalsignaturestoensurethatlookupdataiscorrectandconnectionsarelegitimate.Then,inasecureenvironment,you’resupposedtousemainlyDNSSEC,andthenitisimportanttomonitorthecriticalDNSSECparameters;thoseitemscanberesumed,asfollows:

Thezonefile’svalidityThezones’rolloverstatusTheDNSresponsetime

Currently,therearetwopluginsavailabletoimplementchecksagainsttheDNSSECzonerollover:

RollstateZonestate

Thefirstonechecksthezonemanagedbythedaemonrollerd;thesecondonechecksthevalidityofDNSzones.

NoteThefullcodeisavailableathttps://github.com/hardaker/dnssec-tools/tree/master/dnssec-tools/apps/zabbix,andthepackageisavailableathttp://www.dnssec-tools.org/download/dnssec-tools-2.1.tar.gz.

OneoftherequirementstoproperlysetupthispluginisthatyouneedtobeawareofthefrequencyofyourrolloveractionstotunetheZabbixitem;pleasebeawarethatalittlelatencyisnormalhere.Anyway,aslongasyoudon’trolloverzoneseveryfewminutes(TTLissettoafewminutes),thislagwillnotbeanissue.

Now,beforeyoucanruntheplugin,youneedtohaveinstalledafewrequiredPerlmodules:

#perl-MCPAN-eshell

cpan>installNet::DNS

cpan>installNet::DNS::SEC

Wearesupposingthatyoualreadyhavecpaninstalled;ifyoudon’thaveitinstalledinyoursystem,pleaseinstallitwiththefollowinglineofcode:

#yuminstallcpan

Now,onceyouhaveinstalledtherequiredmodule,youneedtoinstalltheopenssl-develpackagewiththefollowingcommand:

#yuminstallopenssl-devel.x86_64

Now,youcanfinallyuncompressthesoftwarewiththefollowingcode:

www.it-ebooks.info

#tar-zxvf./dnssec-tools-2.1.tar.gz

#cd./dnssec-tools-2.1

#./configure&&make&&makeinstall

Nowin/dnssec-tools-2.1/apps/zabbix/,wehavealltheneededsoftware.Herearethepiecesofsoftwareavailablein/dnssec-tools-2.1/apps/zabbix/:

total40

-rwxrwxr-x.112741274768Jan22013backup-zabbix

-rw-rw-r--.1127412741706Jan22013item.fields

-rw-rw-r--.1127412742878Jan22013README

-rwxrwxr-x.1127412746763Feb152013rollstate

-rwxrwxr-x.1127412747720Feb152013uemstats

-rw-rw-r--.1127412741329Oct192011zabbix_agentd.conf

-rwxrwxr-x.1127412746314Feb152013zonestate

Finally,wecantryournewplugins,asfollows:

#./rollstatemydomain.com

ZSKphase3

#./zonestatemydomain.com

zonefilevalid

Now,it’stimetoenableournewplugins;todothis,weneedtodefineacoupleofnewentriesofUserParameterontheagentside’s/etc/zabbix/zabbix_agentd.conf:

UserParameter=dnssec-tools.rollover.status[*],rollstate$1

UserParameter=dnssec-tools.rollover.statusnum[*],rollstate–numeric$1

Evenhere,youneedtoplacetherollstateplugininadirectorycontainedinthepathorusethefullyqualifiedpathforourplugin.Also,onceyouhaveaddedUserParameter,youneedtorestarttheagentwith:

#servicezabbix-agentrestart

ShuttingdownZabbixagent:[OK]

StartingZabbixagent:[OK]

Therollstatepluginprovidestwodifferentoutputswiththe–numericoptionspecified.ItprovidespositivenumbersfortheZSKphasesandnegativenumbersfortheKSKphases.ThisenablesustoproduceagraphthatrepresentsallthephasesofDNSSEC.

OnceyouhavecreatedtheZabbixagentitemonyourtemplateandyourscriptisrunning,theoutputwillbelikethenextscreenshot.

Intheexampleandtherelativegraph,wehaveahighlyfrequentrollover.Inareal-lifescenario,thetimerequiredtogothroughallthedifferentstatuseswillbelonger.

www.it-ebooks.info

ThedetailsoftheDNSSECrolloverintextmode,usefultokeeptrackofallthestatuschanges,willbecontainedinatextitem.Anexampleofthelatestdataisshowninthenextscreenshot:

Asyoucansee,youwillhaveahistoricalstatusofallthestepscrossedduringtherollover,andyouwillhaveacleartrackofthestepsperformed.

NoteThisitemwillbepreciousifyourprocessgetsstuckonastep,especiallyifthishappensperiodically.

Inthenextscreenshot,youcanseethezonestatuspluginatwork:

www.it-ebooks.info

Now,theonlythingyoustillhavetodoiscreateatriggerbasedontheinformationwe’reacquiring.Here,itisimportanttobearinmindthatalittlelagisnormalduringthezonetransferprocess;thislagneedstobeconsideredwhenyousetupthetrigger.

www.it-ebooks.info

ApachemonitoringMostofthereverseproxiesarenowadaysimplementedusingApache.Apache,otherthanbeingawebserver,isquiteusefulasareverseproxyasitincludessomepowerfulmodules:

mod_proxy

mod_proxy_http

mod_proxy_ftp

Otherthanasareverseproxy,itcanbeusedasaloadbalancerthanksto:

mod_proxy_balancer

Now,unfortunately,thereisn’tavalidmethodtoacquirethemetricsstrictlyrelatedtothemoduleused,butanyway,wecanacquirequiteafewmetricsfromApacheitself.

Thefirstthingyouhavetodobeforeyoucanacquirethestatisticsisenablethem.Todothis,youneedtoputthefollowinglinesinyourApacheconfigurationfile:

SetHandlerserver-status

Allowfrom127.0.0.1

Orderdeny,allow

Denyfromall

</Location>

Also,youcanoptionallyaddthefollowinglinetoyourglobalApacheconfigurationfile:

ExtendedStatusOn

Here,weareconfiguringthemodulewiththeExtendedStatusOnoption.Withthissetting,Apachekeepstrackofextendedstatusinformationforeachrequest.Thiscollectioncanslowdowntheserver,andifyounoticeperformanceissues,itcanbedisabledwiththeExtendedStatusOffkeyword.

TipPleasekeeprestricted,asmuchasyoucan,theaccesstothe/server-statuslocation.Inourcase,itisallowedonlyfrom127.0.0.1.ThismeansthatyouneedtocollectthestatisticsfromtheagentinstalledlocallyonyourApachehost.Itisimportanttoknowthatifmod_statusiscompiledintotheserver,thenitshandlerisavailableinallconfigurationfiles,includingper-directoryfiles,likehtaccess.Thiscanhavesecurity-relatedramificationsforyoursite.

Now,allyouhavetodoisrestartyourApacheandcheckwhetheryoucanretrievethestatisticsrunningthefollowingcommand:

[root@localhost~]#curlhttp://127.0.0.1/server-status

<!DOCTYPEHTMLPUBLIC"-//W3C//DTDHTML3.2Final//EN">

<title>ApacheStatus</title>

</head><body>

www.it-ebooks.info

<h1>ApacheServerStatusfor127.0.0.1</h1>

<dl><dt>ServerVersion:Apache/2.2.15(Unix)DAV/2PHP/5.3.3</dt>

<dt>ServerBuilt:Jul23201414:17:29

</dt></dl><hr/><dl>

<dt>CurrentTime:Monday,03-Nov-201419:48:11PST</dt>

<dt>RestartTime:Monday,03-Nov-201419:48:00PST</dt>

<dt>ParentServerGeneration:0</dt>

<dt>Serveruptime:11seconds</dt>

<dt>Totalaccesses:9-TotalTraffic:0kB</dt>

ThisApachemodule’soutputisreallyfullofusefulinformation;lookingattheoutputindetail,youcanseethatitprovidestheinformationshowninthefollowingscreenshot:

Here,youhaveaviewthatissplitintofourmainsections,whichareasfollows:

TheApacheversiondata,modulestarted,andserverbuilddetailsTheApacheserverstatusthatprovidesyoutheuptime,CPU,numberofaccess,numberofrequest/sec,andsomemoreinformationaboutitsstatusTheApachescoreboardAsectionwithallthedetailsoftheconnectionserved

Here,retrievingthestatisticsisnotaseasyasyouwouldimagine.Thefirstandsecondsectionsarequiteverbose,anditiseasytoextracttherequiredinformationfromthemonceyou’veobtainedthewebpage.ThethirdsectionisalittlemorecomplexasitistheApachescoreboard.ThescoreboardisarepresentationofApache’sworkersandtheirrelativestatus.TheworkersareApache’srequest-handlerstatus.Thekeysusedonthescoreboardarethefollowing:

www.it-ebooks.info

ScoreboardKey:"_"WaitingforConnection,"S"Startingup,"R"Reading

Request,"W"SendingReply,"K"Keepalive(read),"D"DNSLookup,"C"Closing

connection,"L"Logging,"G"Gracefullyfinishing,"I"Idlecleanupof

worker,"."Openslotwithnocurrentprocess

Then,toretrieveandanalyzethestatus,weneedtouseaslightlydifferentURL:http://localhost/server-status?auto.

WecantrytheoutputproducedbythisURLusingcurl,asfollows:

#curlhttp://127.0.0.1/server-status?auto

TotalAccesses:1334

TotalkBytes:2163

CPULoad:5.20713

Uptime:2776

ReqPerSec:.480548

BytesPerSec:797.879

BytesPerReq:1660.35

BusyWorkers:1

IdleWorkers:10

Scoreboard:

_______W___…...............................................................

...........................................................................

.............................

Now,it’seasytoretrievetheCPULoadvalue,forinstance:

#curl-shttp://127.0.0.1/server-status?auto|awk'/^CPULoad:/{print

5.15882

Withthesamemethod,wecanacquireallthemetrics,forexample,thenumberofIdleWorkerswillbe:

#curl-shttp://127.0.0.1/server-status?auto|awk'/^IdleWorkers:/

{print$2}'

Parsingthescoreboardisalittledifferentasweneedtocountthenumberof_ifwearelookingatalltheworkersthatarewaitingforaconnectioninsteadofcountingalltheoccurrencesofWtocheckalltheworkersthataresendingreplies.Toaddressthisrequirement,youcanusethefollowingcommand:

#curl-shttp://127.0.0.1/server-status?auto|awk'/^Scoreboard:/

{print$2}'|awk'BEGIN{FS="_"};{printNF-1}'

ThefirstawkcommandidentifiestheScoreboard:section,thesecondawkcommandcountsalltheoccurrencesof_intheline,definingafieldseparator,andthencountingallthematchedfields.

Currently,therearethreeprebuiltpluginstodothis:

zapache:ThisisashellscriptcalledviaUserParameterZabbixApacheUpdater:ThisisaPythonsoftwarethatneedstobescheduledon

www.it-ebooks.info

crontabquery_apachestats.py:ThisisaPythonsoftwaretriggeredbyUserParameter

Inthissection,wewillanalyzezapacheasitusesthesamemethoddescribedtoacquiremetricsfrommod_statusofApache.Thescriptisavailablefordownloadathttps://github.com/lorf/zapache.

Allyouhavetodoisdownloadzapachefromthatlocation,copyzapacheunder/home/zabbix/bin/withtherelativetemplate,andthenconfigureUserParameterintheagentconfigurationfile/etc/zabbix/zabbix_agentd.conf,asshownhere:

UserParameter=zapache[*],/home/zabbix/bin/zapache$1

Now,ontheGUI,youhavetocreateyourtemplateorimporttheonedistributedwithzapache.Then,navigatetoConfiguration|Template|Importandselectthezapache-template.xmltemplateifyouwanttheitemasZabbixagentorthezapache-template-active.xmltemplateifyouprefertheitemsmanagedasZabbixagent(active).

Ifyoutakealookatthezapachesourcecode,youwillnoticethatitcanrunasZabbixagent’smodeorasanexternalscript,whichmeansthatyoucanuseittoacquiretheApachestatisticslocallyonthesameserverorremotely.

Hereisthecodesectionthatmanagesthiskindofbehavior:

if[[$#==1]];then

#AgentMode

STATUS_URL="http://127.0.0.1/server-status?auto"

CASE_VALUE="$1"

elif[[$#==2]];then

#ExternalScriptMode

STATUS_URL="$1"

case"$STATUS_URL"in

http://*|https://*);;

*)STATUS_URL="http://$STATUS_URL/server-status?auto";;

CASE_VALUE="$2"

Asyoucansee,youcanrunthescriptwithonlyoneparameter,whichrepresentsthemetricyouwouldliketoacquire,ortwoparameters,specifyingeventheremoteIPaddressofyourApachereverseproxyorwebserver.Here,inordertokeepthingseasy,weavoidmod_statusfrombeingaccessedexternallyusingaUserParameter.Anyway,itisbettertobeawarethatyoucanevencentralizestatisticacquisitionthankstothiscodesection.

ThefinalresultofoursetupandApache’smetricacquisitionisshowninthenextscreenshot:

www.it-ebooks.info

Now,itistimetodiscusstriggersrelatedtothisApachemonitoring.Firstofall,youneedtocreateatriggerbasedonthelastvalueofzapacheping,asfollows:

{TemplateAppApacheWebServerzapache:zapache[ping].last(0)}=0

Ofcourse,ifthezapachepingfails,returning0,youhaveanissue.Someotherparametersthatarecriticalforserverstatusandonwhichyoucancreatetriggersare:

WaitingForConnection:ThisindicatesthatthenumberofprocessesarewaitingforaconnectionReqPerSec:ThisindicatesthenumberofrequestspersecondCPULoad:ThisindicatestheamountofCPUconsumedbyApache

Thosevaluesarestrictlydependentontheserveryou’reusing,thenumberofclientsyouareserving,andmostimportantly,whatexactlyandhowyouareservingtherequest.Aboutwhatandhowyouareservingtherequest,youcanhavesomeverycomplexrewritingandreverserulesthatcanmakeagroupofURLsmorecomplextomanage.Here,thebestthingtodoistrytofindoutyourApache’slimitusingsometoolsthatareabletoproducealotofconcurrentconnectionsandthenworkload,forinstance,youcantrySiege.

NoteMoreinformationaboutSiegeisavailablehere:http://www.joedog.org/siege-home/.

Onceyou’vetestedandfoundthemaximumnumberofclientsyoucanserveperURLandyou’veseenthewebserverlimits,youcancreateandtuneyourcustomtriggers.

www.it-ebooks.info

NTPmonitoringThesystemclockissomethingyoushouldkeepmonitoringbecauseif,forsomereason,yoursystemsuffersasystemclockdrift,thiscanbecomeabigissue.

Performingapracticalexampleofheavydriftonthesystemclockwillcauseissues.TheDNSSECzonereplication,yourFTPservice,theIMAPservice,andmanyotherserviceswillbeaffected,makingyourserverunstableandunusable.

TokeepyoursystemclockinsyncwiththeremoteNTP,youcanuseandinstalltheNTPdaemonthatwilltakecareofthesystemclock.

ToinstallNTP,youcanuseyumasusual:

#yuminstallntp

...outputremovedhere…

Installed:

ntp.x86_640:4.2.6p5-1.el6

Complete!

Onceyou’veinstalledtheNTP,youneedtofindtheserverthatisclosertoyouusingthewebsitehttp://www.pool.ntp.org/en/.

Fromthiswebsite,youneedtochoosetheserverthatisbetterforyouandthenchangethe/etc/ntp.confconfigurationfile.

Also,itisagoodpracticetoaddthelogfiledirectiveattheendofthentp.confconfigurationfile,asfollows:

#echo"logfile/var/log/ntp.log">>/etc/ntp.conf

Thenstartorrestarttheservice,asfollows:

#servicentpdstop

Shuttingdownntpd:[OK]

#servicentpdstart

Startingntpd:[OK]

Now,youneedtoconsiderthatyoucanhaveonecentralserverusedasaprimaryntpdserverforyournetworkandpropagatethesystemtimefromthere;inthiscase,youneedtochangethe/etc/ntp.confconfigurationfileabit:

#Hostsonlocalnetworkarelessrestricted.

restrict192.168.1.0mask255.255.255.0nomodifynotrap

Nowfinally,youcanattachallthehostsofyournetworktothisntpdserverandthenmonitorthisNTPandtheclient’stime.

TipIfyouareprotectingaserverwithafirewall,youneedtoenabletheUDPonport123onbothdirections.Ifyou’reusingiptablestoenabletheclientandtheservercommunication,youneedtoaddthefollowingrulestotheOUTPUTandINPUTchains:

www.it-ebooks.info

iptables-AINPUT-pudp--dport123-jACCEPT

iptables-AOUTPUT-pudp--sport123-jACCEPT

Now,toretrievemetrics,weneedtoqueryntpd.Forthisoperation,wecanusentpq,whichwillshowallthestatistics.Fromamonitoringperspective,we’relookingfortheoffset,jitter,anddelay.

Inthenextexample,weseethecompleteoutputofntpq,asfollows:

#ntpq-pn127.0.0.1

Remoterefidsttwhenpollreachdelayoffsetjitter

==================================================================

+91.247.253.152191.241.139.1373u964135.27629.4929.791

+217.147.208.1194.242.34.1492u864119.61730.91211.497

*192.33.214.47129.194.21.1952u764125.58132.15711.007

+195.141.190.190212.161.179.1382u664120.73931.14310.983

Pleasenotethatthisserverissufferingabigdriftandthetriggerisalreadyonfire.

Toacquirethemetricthen,wecanuseacommandlikethisone:

#ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{offset=0}$1~/\*/{

offset=$9}END{printoffset}'

32.157

ThiscommandretrievestheoffsetbetweenthesystemclockandtheNTPserver.

NoteWeareusingthe–pand–noptionstogether;withthe–noption,weareavoidingthenameresolution,andthentheDNSquery.Thisisdoneinordertokeeptheitemaslightweightaswecan.

Now,wecanquicklysetupNTPmonitoringusingUserParameterontheagentsidewith:

UserParameter=ntp.jitter,ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{

offset=0}$1~/\*/{offset=$9}END{printoffset}'

ThiswillsetUserParametertoretrievethejittervalue;anyway,wecanevendosomethingalittlemorecomplexandthenproduceascriptlikethefollowing:

#!/bin/bash

VERSION="1.0"

functionusage()

echo"ntpcheckversion:$VERSION"

echo"usage:"

echo"$0jitter-Checkntpjitterdelay"

echo"$0offset-Checkntpoffset"

echo"$0delay-Checkntpdelay"

########

#Main#

########

if[[$#!=1]];then

#NoParameter

www.it-ebooks.info

case"$1"in

'jitter')

value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{jitter=0}$1

~/\*/{jitter=$10}END{printjitter}''"

rval=$?;;

'offset')

value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{offset=0}$1

~/\*/{offset=$9}END{printoffset}''"

rval=$?;;

'delay')

value="'ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{delay=0}$1

~/\*/{delay=$8}END{printdelay}''"

rval=$?;;

exit1;;

if["$rval"-eq0-a-z"$value"];then

rval=1

if["$rval"-ne0];then

echo"ZBX_NOTSUPPORTED"

echo$value

Then,ontheagentside,wecandeploythisscriptcalledntpcheck.shinthe/home/zabbix/bindirectory:

#ls-la/home/zabbix/bin/ntpcheck.sh

-rwxr-xr-x1zabbixzabbix781Nov903:23/home/zabbix/bin/ntpcheck.sh

Oncethisisdone,allwehavetodoiscreateUserParameter,asfollows:

UserParameter=ntp[*],/home/zabbix/bin/ntpcheck.sh$1

Then,restarttheagent:

#servicezabbix-agentrestart

ShuttingdownZabbixagent:[OK]

StartingZabbixagent:[OK]

Testournewitems:

#zabbix_get-s127.0.0.1-kntp[jitter]

#zabbix_get-s127.0.0.1-kntp[offset]

-6.696

#zabbix_get-s127.0.0.1-kntp[delay]

18.956

Andintheend,createourthreenewitemsontheZabbixGUI,asshowninthefollowingscreenshot:

www.it-ebooks.info

NTP–whatarewemonitoring?Now,evenifthoseitemnamesappearassomethingeasytounderstand,itisbettertoknowwhatwearemonitoring.Firstofall,weneedtoclarifythatwe’reacquiringvaluesforthecurrenttimesource,hencewearetakingthevaluesinthelinethatbeginswitha*fromthentpqoutput.Forconvenience,thentpqoutputisreportedhere:

#ntpq-pn127.0.0.1

Remoterefidsttwhenpollreachdelayoffsetjitter

==================================================================

+91.247.253.152191.241.139.1373u964135.27629.4929.791

+217.147.208.1194.242.34.1492u864119.61730.91211.497

*192.33.214.47129.194.21.1952u764125.58132.15711.007

+195.141.190.190212.161.179.1382u664120.73931.14310.983

Asyoucansee,thelinesofthisoutputarenotordered,andtheybeginwith+and*(inthisexample).Weareinterestedintheonethatbeginswith*.Thereasonisthatthelinethatbeginswith*representsthepreferredandcurrenttimesource.

Wecanevenhaveaprefixlikethefollowing:

+:Thissignindicatesthatthepeerisagood,preferredremotepeerorserver(space),x,-,#,and.:Theseindicatethatthispeerisnotbeingusedforsynchronization

Now,wehaveclarifiedthereasonwhywearerunningthisawkcommand:

#ntpq-pn127.0.0.1|/usr/bin/awk'BEGIN{delay=0}$1~/\*/{delay=$8

}END{printdelay}'

Now,tohavesomemoredetailsaboutwhatwe’reacquiring,wecandefinethemas:

Delay:Thisisthecurrentestimateddelay.Itisthetransittimebetweenremotepeersorserversinmilliseconds.Offset:Thisisthecurrentestimatedoffset.Itisthetimedifferencebetweenremotepeersinmilliseconds.Jitter:Thisisthecurrentestimateddispersion,orbetter,thevariationindelaybetweenthesepeersinmilliseconds.

NoteIfyou’remonitoringaserverthatisrunninginavirtualenvironment,youneedtobeawarethatpracticallyallthevirtualizationsoftwaresuffersfromsystemclockdrift.Thencheckthevendor-specificbestpracticetoreducetheNTPdrift.

Nowit’stimetochangethescriptalittleaswecanchecktheNTPhealthstatusbyaddingthefollowingcasestatement:

case"$1"in

'health')

primary="'ntpq-pn127.0.01|grep^\*|grep-vgrep|wc-l'"

rval=$?

www.it-ebooks.info

if["${primary}"-eq"1"];then

value="1"

value="0"

Now,wecancheckwhetherwehaveatleastoneprimarypreferredsourcedefinedtogettheNTPsyncinagoodshape.Weneedtothenaddanewitemandarelatedtriggerthatwillgoonfireifthevaluereturnedis0.Otherthanthistrigger,wecanevenhaveatriggerthatwillgoonfireiftheclockdriftisbiggerthan50millisecondsforinstance,orevenless.

Inthenextscreenshot,youseetheinteractionbetweentheJitter,Offset,andDelayonaLinuxvirtualserver(thatsufferfrombigsystemclockdrifts):

www.it-ebooks.info

SquidmonitoringSquidisthemostdiffusedcachingproxyfortheWeb.SquidsupportsHTTP,HTTPS,FTP,andmanymoreprotocols.Thisproxysoftwarereducesalotofthebandwidthrequiredtoserveitsclientsandimprovestheresponsetime,implementingaverygoodcachingsystem.Forallthosereasons,itisquiteevidentwhyyoushouldhaveSquidtomonitorinsideyournetwork.

TherearetwoprimarywaystoacquiredataandmetricsfromSquid:

UsingSNMPUsingsquidclient

Ifyou’recuriousabouttheSNMPsetupontheSquidserver,youcanhavealookattheofficialdocumentation,inparticularthesectionavailableathttp://wiki.squid-cache.org/Features/Snmp.

WeshouldavoidenablingSNMPonourSquidasithasbeenaffectedinthepastbymanyoverflowsandissues.Thelastsecurityissue,atthetimeofwritingthis,causedbySNMPenabledonSquid,isavailableathttp://www.squid-cache.org/Advisories/SQUID-2014_3.txt,andasyoucansee,itisareallyrecentissue.

Fortunately,theclientisreallypowerfulandthispermitsustoimplementagoodmonitoringsolutionwithoutenablingSNMP.

Typethefollowingcommand:

#squidclientmgr:info

Inresponsetotheprecedingcommand,Squidwillprintouttheentirestatisticdomainacquireduntilnow:

HTTP/1.0200OK

Server:squid/3.1.10

Mime-Version:1.0

Date:Sun,09Nov201417:23:25GMT

Content-Type:text/plain

Expires:Sun,09Nov201417:23:25GMT

Last-Modified:Sun,09Nov201417:23:25GMT

X-Cache:MISSfromlocalhost.localdomain

X-Cache-Lookup:MISSfromlocalhost.localdomain:3128

Via:1.0localhost.localdomain(squid/3.1.10)

Connection:close

Then,asyoucanunderstand,itwillbequiteeasytoretrievesomeimportantitemsfromthiskindofoutput.Tryingoutanexample,ifyouwouldliketoacquiretheCPUUsage,youcansimplyrun:

#squidclientmgr:info|grep'CPUUsage:'

CPUUsage:0.01%

Ofcourse,thiskindofoutputneedstobealittleshapedtobeusableforourwork,thenext

www.it-ebooks.info

commandwillbeaUserParameterreadycommand:

#squidclientmgr:info|grep'CPUUsage:'|cut-d':'-f2|tr-d'%'|tr-d'

Now,wehavetwowaysofdoingthis:

WecreatealonglistofUserParameterontheagentsideWecreatejustaone-userUserParameterandcallitusingaparameter

Thesecondwayisthepreferredapproachasifyouneedtoaddanitemtoacquire,youdon’tneedtorestarttheagent.Hereduetospaceconstraints,wewillnotcommentallthescript;forthecompletescript,pleaserefertoAppendixB,CollectingSquidMetrics.

YouneedtocreateUserParameter:

UserParameter=squid[*],/home/zabbix/bin/squidcheck.sh$1

Now,youneedtorestarttheagent,andyoucancheckwhetheryou’reabletoacquirethemetricswiththefollowingcommand:

#zabbix_get-s127.0.0.1-ksquid[icp_sent]

Ifyoucanretrievethemetrics,theconfigurationisfine.

Now,ontheserverside,youneedtocreateyouritems,asshowninthefollowingscreenshot:

Nowthatwearefinallyacquiringallthemetrics,itisimportanttodefineatleasttwo

www.it-ebooks.info

triggers:

OnetiedtothenumberofSquidprocessesrunningthatshouldneverbe0Onetiedtothenumberofavailablefiledescriptors;ifthisnumberislessthan100,weneedtohaveatriggeronfire

Thisisshowninthefollowingscreenshotandistheminimumnumberoftriggersyoushouldhave:

ToclosetheSquidmonitoring,wecantellthatyouarenowabletoacquireatleast22itemsusingthescriptavailableonGitHubathttps://github.com/smartmarmot/zabbix_network_monitoring/tree/master/Chapter3;youcannowsetmanyothertriggersdependingonyoursetup,servercapacity,numberofclientstoserve,andthemeanofthenumberofpagesrequiredbyyourclientnetwork.

Amongthemostimportantparameterstomonitor,wehave:

Thebytehitratioover5and60minutesTherequestdiskhitratioover5and60minutesRequestfailureratio

Allthehitratiosneedtobeascloseto100percentaspossible.Everyvalueofcachingunder70percentshouldmakeatriggergoonfire,andeventherequestfailureratio,ifitishigherthan30,shouldtriggeranalarmasitistellingusthatoursystemisnotrespondingproperly.

www.it-ebooks.info

SummaryInthischapter,wecoveredalargenumberofcomponents.Westartedourdiscussionfromthemostusedandevenverycriticalnetworkservice:DNS.Goingaheadonthesameway,wediscussedDNSSEC;then,wemovedontoApache,themostusedandeffectivereverseproxy;walkedthroughNTP;andclosedthechapterwithSquid,themostinstalledandusedproxyservice.Forallthesystemsandservicesanalyzed,you’renowabletoacquirethemostcriticalmetrics,andyouknowhowtocreateeffectivetriggers.

Triggersherearecoveringthemostcriticalroleandhenceyourexperiencewithinyournetworkisthetrulyaddedvalue.You,withtheknowledgeacquiredfromthischapterandyourenvironmentexperience,willbethekeytocreatingeffectiveandproactivetriggers.Thischapterhascoveredallthecriticalservicesyoucanfindinanetwork,andnowyoucaneasilyprovideaheavyaddedvalue,creatingproactivechecksandinstallinganeffective,tailor-mademonitoringsolution.Inthenextchapter,youwilllearnhowtoautomatethediscoveringofyournetwork’selementsandhowtoapplyatemplatetothediscovereditem.Also,youhavetoadaptyourmonitoringsystemwithinyourenvironments,andthiskindoftaskisthetypicalboringandtime-consumingtaskthatanetworkadmindoesn’tliketodo.Thechapterwillprovideyouwithallthenecessaryinformationtousethehostdiscoveryandthelow-leveldiscoveryinaneffectiveway.Youwillbeguidedthroughthedifficultwaytoautomatetheitemdiscovery:thiswillheavilyreducethetimeneededtostartupyourmonitoringsolutionbutwillimpactandreducethetimeneededtomaintainyourgrowinganddynamicallymovingsetup.

www.it-ebooks.info

Chapter4.DiscoveringYourNetworkInthepreviouschapters,we’veseenhowtogetdifferentmetricsfromquiteafewdifferentsources,usingdifferentmethods.Whatwehaven’tcoveredyet,ishowtoeasilygetallthisdataintoZabbixwhenyouhaveagreatnumberofmonitoredobjects.

Manuallycreatinghosts,items,andtriggersisanexcellentexercisetogetthehangofhowthingsworkinZabbix,butitcanquicklybecomearepetitive,boring,error-proneactivity.Inotherwords,theyarethekindsoftaskscomputersweremadeforinthefirstplace.

Whatifyourmonitoringsolutioncouldjustfindthehostsanddevicesyouwanttomonitor,addthemasZabbixhosts,applyatemplate,andstartmonitoringthem?Andwhatifitdidn’tjustlimititselftofindinghoststomonitor,butitalsofoundoutwhetheryourswitchhas24or48ports,howmanydisksyourwebserverhasattached,andwhatportsareopenonacertainhost?Aftersomeinitialconfiguration,youwouldnothavetobotherwithaddingorremovingthingstomonitor.Itwouldcertainlybegreat,buttheproblemwithautomateddiscoveryisthatitoftenhastocometotermswiththerealityofareal-worldnetwork,whichisoftenfullofexceptionsandspecialrules.Insuchcases,youcouldfindyourselfspendingalotoftimetryingtoadaptyourmonitoringsystemtoyourenvironmentinordertocatchupwithanautomateddiscoverythatmightbejustalittletooautomatic.

Luckily,Zabbixcansupportmanydifferentdiscoverystrategies,mixthemupwithregularhostanditemcreation,andgenerallyprovideagoodbalancebetweentheneedtohaveafullyautomatedsystemandtheneeddohaveamonitoringsolutionthatmatchesascloselyaspossibletheenvironmentithastomonitor,withallitsexceptionsandspecialcasesthatareimpossibletocapturewithjustadiscoverystrategy.

ThischapterwillbedividedintotwomainpartsthatmirrorthetwomainlevelsofdiscoverythatZabbixsupports:networkdiscoveryandlow-leveldiscovery.Theformerisusedtofindoutwhichhostsareinyournetwork,andthelatterisusedtofindoutwhatfacilitiesandcomponentsarefeaturedinagivenhost.

Let’sstartwithfindingouthownetworkdiscoveryworksandhowtomakethemostoutofit.

www.it-ebooks.info

FindinghoststheZabbixwayZabbix’sdiscoveryfacilitiesconsistofasetofrulesthatperiodicallyscanthenetwork,lookingfornewhosts,ordisappearingones,accordingtopredeterminedconditions.

ThethreemethodsZabbixcanusetocheckfornewordisappearedhosts,givenanIPrange,are:

TheavailabilityofaZabbixagentTheavailabilityofanSNMPagentTheresponsetosimpleexternalchecks(FTP,SSH,andsoon)

Thesecheckscanalsobecombined,asillustratedinthefollowingexample:

Asyoucansee,whenenabled,thisrulewillcheckeveryhour,intheIPrange192.168.1.1-254,foranyserverthat:

ReturnsanSNMPv3valuefortheSNMPv2-MIB::sysDescr.0OIDIslisteningtoandacceptingconnectionsviaSSHHasanHTTPSserverlisteningonport8000

Beawarethatadiscoveryeventwillbegeneratedifanyoneoftheseconditionsismet.

www.it-ebooks.info

So,ifadiscoveryrulehasthreechecksdefinedandahostinthenetworkrespondstoallthreechecks,threeeventswillbegenerated,oneperservice.

AsusualwithallthingsZabbix,adiscoveryrulewillnotdoanythingbyitself,exceptgenerateadiscoveryevent.ItwillthenbethejobofZabbix’sactionsfacilitytodetecttheaforesaideventanddecidewhetherandhowtoactonit.

Discoveryeventactionsareverysimilartoregulartriggereventactions,soyou’llprobablybealreadyabletomakethemostoutofthem.ThemainthingtorememberisthatwithZabbix,youcannotactdirectlyonaneventtocreateordisableahost:youneedtoeithercopytheeventdatabyhandsomewhereandthenproceedwithallthemanualoperationsneededbasedonthatdata,oryouneedtoproperlyconfiguresomeactionstodothatworkforyou.Inotherwords,withoutaproperlyconfiguredaction,adiscoveryrulewillnotaddbyitselfanydiscoveredhosttothelistofmonitoredones.

Everyactionhasaglobalscope:it’snottiedtoanyparticulartrigger,host,orhostgroupbydefault.Thismeansthatwhenyoucreateanaction,you’llneedtoprovidesomeactionconditionsinordertomakeitvalidonlyforcertaineventsandnotothers.ToaccessthediscoveryactionssectioninthewebUI,headtoConfiguration|ActionsandthenselectDiscoveryfromtheEventsourcedrop-downmenu,justundertheCreateactionbutton.

Whenyoucreateanaction,you’llstartwithgivingitanameanddefiningadefaultmessageintheactiondefinitionsection.You’llthenmovetotheactionconditionssectiontoprovidefilteringintelligence,beforefinishingwiththeactionoperationssectiontoprovidetheaction’scorefunctionality.Actiondefinitionsareprettysimpleasyou’lljustneedtoprovideauniquenamefortheactionandadefaultmessage,ifyouneedone.So,let’smovestraighttotheinterestingsectionsofactionconfiguration:conditionsandoperations.

www.it-ebooks.info

DefiningactionconditionsTheactionconditionssectionletsyoudefineconditionsbasedontheevent’sreportedhostIPaddress,servicestatusandreportedvalue,discoveryrules,andafewothers:

TheReceivedvalueconditionisofparticularinterest,asitallowsyoutodothingslikedifferentiatingbetweenoperatingsystems,applicationversions,andanyotherinformationyoucouldgetfromaZabbixorSNMPagentquery.Thiswillbeinvaluablewhendefiningactionoperations,asyou’llseeinthenextparagraph.Areceivedvaluedependsonthediscoveryruleandontheoutputofthediscoveryeventthattriggerstheaction.Forexample,ifadiscoveryruleissettolookforhostsrespondingtoanSNMPGetfortheSNMPv2-MIB::sysDescr.0OID,andthatrulefindsarouterthathasC3745asthevalueofthatOID,thenthediscoveryeventwillpassC3745totheactionasthereceivedvalue.

Singleconditionscanbecombinedtogetherwithlogicaloperators.There’snotmuchflexibilityinhowyoucancombinethemthough.

YoucaneitherhaveallAND,allOR,oracombinationofthetwowhereconditionsofdifferenttypesarecombinedwithAND,whileconditionsofthesametypearecombinedwithOR.

www.it-ebooks.info

ChoosingactionoperationsDiscoveryactionsaresomewhatsimplerthantriggeractionsastherearenostepsorescalationsinvolved.Thisdoesn’tmeanthatyoudon’thavequiteafewoptionstochoosefrom:

Pleasenotethatevenifyoudefinedadefaultmessage,itwon’tbesentuntilyouspecifytherecipientsinthissectionusingtheSendmessageoperation.Ontheotherhand,ifadding(orremoving)ahostisaquiteself-explanatoryaction,whenitcomestoaddingtoahostgrouporlinkingtoatemplate,itbecomesclearthatagoodsetofactionswithspecificreceivedvalueconditionsandtemplate-linkingoperationscangiveahighlevelofautomationtoyourZabbixinstallation.

NoteThishighlevelofautomationisprobablymoreusefulinrapidlychangingenvironmentsthatstilldisplayagoodlevelofpredictability,forexample,thekindofhostsyoucanfind,suchasfast-growinggridsorclusters.Inthesekindsofenvironments,youcanhavenewhostsappearingonadailybasis,andmaybeoldhostsdisappearatalmostthesamerate,butthekindofhostismoreorlessalwaysthesame.Thisistheidealpremiseforasmallsetofwell-configureddiscoveryrulesandactions,soyoudon’thavetoconstantlyandmanuallyaddorremovethesametypesofhosts.Ontheotherhand,ifyourenvironmentisquitestableoryouhaveaveryhighhosttypevariability,youmightwanttolookmorecloselyatwhich,andhowmanyhosts,youaremonitoringasanyerrorcanbemuchmorecriticalinsuchenvironments.

Also,limitingdiscoveryactionstosendingmessagesaboutdiscoveredhostscanprove

www.it-ebooks.info

quiteusefulinsuchchaoticenvironmentsorwhereyoudon’tcontroldirectlyyoursystems’inventoryanddeployment.Insuchcases,gettingsimplealertsaboutnewhosts,ordisappearingones,canhelpthemonitoringteamkeepZabbixupdateddespiteanycommunicationfailurebetweenITdepartments,accidentalorotherwise.

Moreover,youarenotstuckwithe-mailsandSMSesfornotificationsorlogging.InanActionoperationform,youcanonlychooserecipientsasZabbixusersandgroups.Iftheusersdon’thaveanymediadefined,ortheydon’thavetherightmediafortheactionoperation,theywon’treceiveanymessage.AddingmediatousersisdonethroughtheAdministrationtaboftheZabbixfrontend,whereyoucanalsospecifyatimewindowforaspecificmediatobeused(sothatyouwon’tgetdiscoverymessagesasanSMSinthemiddleofthenightforexample).Speakingofusersandmediatypes,youcanalsodefinecustomones,throughtheMediatypessectionoftheAdministrationtabinZabbix’sfrontend.NewmediatypeswillbeavailablebothintheMediasectionoftheuserconfigurationandastargetsformessagesendingintheActionoperationsform.

AninterestingusefornewmediatypesistodefinecustomscriptsthatcangobeyondsimpleemailorSMSsending.

AcustommediascripthastoresideontheZabbixserver,inthedirectoryindicatedbytheAlertScriptsPathvariable,inthezabbix_server.confconfigurationfile.Whencalledupon,itwillbeexecutedwiththreeparameterspassedbytheserverandtakenfromtheactionconfigurationinthecontextoftheeventthatwasgenerated:

$1:Thisistherecipientofthemessage$2:Thisisthesubjectofthemessage$3:Thisisthemainmessagebody

Therecipient’saddresswillbetheonedefinedforthenewmediatypeinthecorrespondingmediapropertyfortheuserspecifiedintheactionoperationstep.Thesubjectandthemessagebodywillalsobepassedaccordingtotheactionoperationstep,asshownintheprecedinglist.ThisisallthatZabbixneedstoknowaboutthescript.

Thefactis,acustomscriptcanactuallydomanydifferentthingswiththemessage:loggingtoalocalorremotedirectory,creatinganXMLdocumentandinteractingwithalogmanagerwebservicesAPI,printingonacustomdisplay—justaswitheverycustomsolution,thesky’sthelimitwithcustommediatypes.

Hereisasimple,practicalexampleofsuchacustommediatype.Let’ssaythatyourITdepartmenthasimplementedaself-provisioningserviceforvirtualmachinessothatdevelopersandsystemadminscancreatetheirownVMsandusethemforalimitedamountoftimebeforetheyaredestroyedandtheresourcesrecycled.Thislaboratoryofsortshasbeenputinaseparatenetwork,butusersstillhavetogainaccesstoit,andtheyarealsoadministratorsofthoseVMs,sothere’sverylittlecontroloverwhatgetsinstalled,configured,oruninstalledonthosemachines.Inotherwords,whileyoucouldprovisiontheVMswithapreinstalledZabbixagent,youcan’treallyrelyonthefactthatyourusers,whetherinadvertentlyorforspecificreasons,wouldnotdisableit,orwouldnotinstallservicesthatshouldreallynotbethere,likeaDHCPserverforexample.So,youdecideto

www.it-ebooks.info

keepaneyeonthosemachinesdirectlyfromtheZabbixserver(orasuitableproxy)andimplementasimplediscoveryrulethatwillgenerateadiscoveryeventforeveryhostthatrespondstoanICMPechorequestandnothingmore,asfollows:

Basedonthatrule,you’llwanttoconfigureanactionthat,foreveryhostinthatsubnet,willperformaportscanandreporttheresultsviamailtoyou.

Todothat,you’llfirstneedtohaveacustommediatypeandthecorrespondingscript.So,youheadtoAdministration|MediatypesandclickonCreatemediatype.Oncethere,youassignasuitablename,selectScriptasatypeandprovideZabbixwiththenameofthescripttoexecute.Here,youjustneedtodefinethescriptname,asshowninthefollowingscreenshot.You’llfindoutlaterinthechapterinwhatdirectorytheactualscriptshouldbeplaced:

Justaddingamediatypeisnotenoughthough,you’llhavetoenableitfortheuseryouintendtosendthosereportsto.JustheadtoAdministration|Usersandselecttheuseryouwanttoaddthenewmediatypeto.Quitepredictably,thetabyouwantiscalledMedia.Addthemediayoujustcreatedandremembertoalsoaddawaytotellthescript

www.it-ebooks.info

whereitshouldsendtheresults.Sinceyouareinterestedinreceivingane-mailaddressafterall,that’swhatwe’lltellZabbix,asfollows:

TheSendtoparameterwillbethefirstargumentpassedtoport_scan.sh,followedbythesubjectandthebodyofthemessagetosend.So,beforeactuallydeployingthescript,let’sdefinethesubjectandthebodyofthemessage.Todothat,you’llneedtocreateanactionforthediscoveryevent,asfollows:

Forthepurposesofthescript,allyoureallyneedistheIPaddressofthehostyouaregoingtoscan,butitcertainlywouldn’thurttoaddsomemoreinformationinthefinalmessage.

www.it-ebooks.info

Thenextstepistodefinesomeconditionsfortheaction.Rememberthatactionsareglobal,sothefirstconditionyouwanttosetistheIPrangeonwhichthisactionwillbeperformed,otherwiseyou’druntheriskofperformingaportscanoneverydiscoveredhostinyournetwork.

Youmightalsowanttolimittheactionasaconsequenceforthediscoveryruleyoucreated,independentofanyotherrulesyoumighthaveonthesamenetwork.

Finally,youshouldmakeadecisionaboutthediscoverystatus.Ifyouwantaperiodicupdateofwhatportsareopenonadiscoveredhost,you’llalsoneedtodefineaconditionforthehosttobeUp:inotherwords,forthehosttobereportedasliveforatleasttwoconsecutivechecks.

Foraslongasthehoststaysup,aportscanwillbeexecutedandreportedaccordingtothediscoveryintervaloftheruleyoudefinedearlier.Ifyoujustwantaportscanforanewhostorforahostthathasbeenreportedasdownforawhile,you’lljustneedtofiretheactionontheconditionthatthehostisDiscovered;thatis,itisnowbeingreportedup,whileitwasdownbefore.Whatiscertainisthatyou’llwanttoavoidanyactionifthehostisdownorunavailable.

Thefollowingscreenshotencapsulatesthediscussioninthisparagraph:

Thelaststepistodefinetheactionoperationthatissendingthemessageviatheport_scancustommediatypetotheuseryouwant,asfollows:

www.it-ebooks.info

Oncedonewiththis,youarefinallyreadytocreatetheport_scan.shscript.So,headtotheAlertScriptsPathdirectoryasconfiguredinyourzabbix_server.conf(it’susuallydefinedas/usr/lib/zabbix/alertscripts)andcreatethefollowingscriptthere:

#!/bin/bash

RECIPIENT=$1

IPADDRESS=$2

MESSAGE=$3

SCAN="nmap-AT5-sT"

RESULT=$($SCAN$IPADDRESS)

(echo"ScanresultsforIP$IPADDRESS";

echo"$RESULT";

echo"";

echo"$MESSAGE")|mailx-s"Scanresultsfor$IPADDRESS"$RECIPIENT

NoteDon’tforgettosetthecorrectownershipandpermissionsforthescriptonceyouaredone:

#chownzabbixport_scan.sh

#chmod755port_scan.sh

Asyoucansee,theprogramthatwillperformtheactualportscanisNmap,somakesureyouhaveitinstalled.Incaseyoudon’thaveitinstalled,asimpleyuminstallnmapwilltakecareofthat.TheoptionspassedtoNmaparejustthebasics:-sTperformsasimpleconnect()scan.It’snotthefanciestone,butit’stheonlyoneavailabletonon-rootusers,

www.it-ebooks.info

andthescriptwillbeexecutedbyZabbixasthezabbixuser.–Aturnsontraceroute,OS,andservicedetectionsothattheoutputisascompleteaspossible.Finally,-T5forcesNmaptoexecutetheportscaninaslittletimeaspossible.Oncethescripthastheresultsoftheportscan,itwilljustconstructthemessageandsendittotherecipientdefinedintheaction.

Thisis,ofcourse,averybasicscript,butitwillgetthejobdone,andyou’llsoonreceiveaportscanreportforeverynewVMcreatedinyourself-provisioninglab.Tokeepthingssimpleandclear,wedidnotincludeanyconsistencycheckingorerrorreportingincaseofproblems,sothat’scertainlyawayyoucanimproveonthisexample.Youcouldalsotrytosendtheresultstoalogfile(oralogdirectory)insteadofamailaddress,oreventoadatabase,sothatotherautomationcomponentscanpickupthereportsandmakethemavailableviaothermediasuchaswebpages.Whatyou’llprobablywanttoavoidistodirectlychangethehost’sconfiguration,orZabbix’sownone,throughthisscript.

Evenifnoonewillpreventyoufromdoingso,it’sprobablybestifyouavoidusingallthispowertoexecutecomplexscriptsthatmightchangeyournetworkconfiguration,suchasenablinginterfaces,addingrulestoafirewall,andsuchlike.Whilethisisperfectlypossibleusingacustommediascript,thisshouldbethedomainofremotecommands.Thesewilltakecenterstageinthenextparagraph.

RemotecommandsTherearequiteafewoptionsavailabletoyouwhenitcomestoexecutingremotecommandsasanactionoperation.

YoucandefinealistofIPMIcommandstoberunonthetargethostoraseriesofSSHcommandsthatconnecttoaboxandperformvariousoperationsthere.AremotecommandcouldevenbeasimplewrapperforaremotescriptdeployedonaZabbixagent,oracustomscriptthatwillberuneitheronanagentorontheZabbixserveritself.

Thetruthis,sometimes,remotecommandscanbejustalittletoopowerful.Youcanstartandstopservices,deployorprovisionsoftware,makeconfigurationchanges,openorclosefirewallports,andeverythingelseyoucanpossiblyimagine,aslongasyoucanwriteascriptforit.Whilethiscansoundfascinatingandpromising,wehavefoundovertheyearsthatthesesolutionstendtobefragileandunpredictable.OneofthereasonsisthatZabbixdoesn’twarnyouifaremotecommandfails.Moreimportantly,environmentstendtochangefasterthantheseautomationtoolssothatyoucanquicklyfindyourselfdealingwiththeunintendedconsequencesofaremotecommandrunningwhereitshouldnotrun,ornotrunningwhenitshouldrun.

Themoreoftheseyouadd,themoreitwillbehardtokeeptrackofthem,andthemoreonecanbeluredintoafalsesenseofsecurity,countingonthefactthatremotecommandsaretakingcareofthings,while,infact,theymaybecontributingtothechaosinsteadoftamingit.

Thatsaid,it’scertainlyundeniablethatremotecommandscanbeuseful.Let’sseeanexamplethatisbothhelpfulforyourZabbixconfigurationandalsofairlysafe.

www.it-ebooks.info

InChapter2,ActiveMonitoringofYourDevices,we’veseenhowit’spossibletousesomeofthemeasurements,asreportedbyahost’sitems,topopulatethesamehost’sinventoryfields.Thisisagreatsolutionforthefieldsthatcanbefilledthisway,butwhatabouttheotherones?ThingslikePOCdetails,maintenancedates,installername,installedsoftware,andsuchlikecan’talwaysbeextrapolatedfrommonitoringmetricsastheymaysimplynotbeavailableonthemonitoredhostitself.

Theyusuallyareavailable,though,onassetinventorysystemsthatITdepartmentsusetokeeptrackofavailableresources.

Inthefollowingexample,you’llcreateanactionoperationthatwillexecutearemotecommandontheZabbixserver,fetchsomeinventoryinformationfromanassetdatabase,andfilluporupdatethehost’sinventorydetails.

Beforeproceedingwiththecommand,let’smakeanassumptionandsomepreparations.

Therearemanyassetinventorysystemsavailable,someproprietaryandsomeopensource.Allofthemhavedifferentdatabaseschemasanddifferentwaystoexposetheirdata.Moreover,aninventorydatabasestructuredependsasmuchontheactualenvironmentit’sputinto,andtheprocessesthatgoverntheaforesaidenvironment,asitisonitsinternalspecifications.So,wedecidedtouseadummyassetmanagementtoolthatwillreturn,givenanIPaddress,asimpleJSONobjectcontainingalltheinventorydatayouneedforthetaskathand.Theassumptionisthatyou’llbeabletoputtheexampleintoyourcontextandfigureouthowtoextractthesameinformationfromyourowninventorymanagementsystem,andthatyouwillalsoknowwhatauthenticationschemeyouwillrelyonifyouneedtomakejustonerequestormultiplerelatedrequests,andsoon.

Secondly,forpracticalreasonswearegoingtousePythonasthelanguageofthecommandscript,soyou’llwanttomakesurethatit’sinstalledandavailableonyourZabbixserver.Ifit’snotthere,youcaninstallit,andtherelatedutilities,quiteeasilyusingyum:

#yuminstallpython

#yuminstallpython-setuptools

#easy_installpip

Finally,wearegoingtointeractwithZabbix’sconfigurationnotthroughdirectqueriestoitsdatabase,butthroughitsAPI.Inordertodothat,we’lluseaveryusefulPythonlibrary,calledpyzabbix.Youcanfinditathttps://github.com/lukecyca/pyzabbix,butsinceyouinstalledpip,itwillbeextremelyeasytomakeitavailabletoyourPythoninstallation.Justrunthefollowingcommand:

#pipinstallpyzabbix

ThePythonpackagemanagerwilldownloadandinstallitforyou.

Nowwearereadytoconfigurethediscoveryactionandwritetheactualcommandscript.

Youcanchoosetoreuseanexistingdiscoveryrule,suchasthesimpleICMPruleyouusedinthepreviousparagraph,youcancreateanewonespecifictoasinglenetworktoscan,asingleTCPportthathastobeavailable,orthepresenceofaZabbixagent.Wewon’tgo

www.it-ebooks.info

intoanymoredetailshere,asyou’vealreadylearnedhowtoconfigureoneearlierinthechapter.Similarly,wecansafelyskipanydetailabouttheactionconditionsastheymightalsobeentirelysimilartothoseshownearlier.Whatchangesis,ofcourse,theactionoperation.Thefollowingscreenshotwillgiveyouabetterideaofwhatwehavebeentalkingaboutinthisparagraph:

TheimportantelementsherearethefactthatthescriptshouldbeexecutedontheZabbixserver,thefactthatwespecifiedthefullpathforthescript,andthefactthatweareusingthe{DISCOVERY.IPADDRESS}macroastheargument.

Oncetheactionisconfigured,youarereadytopreparetheactualscript.Let’sseehowitwouldlook:

#!/usr/bin/python

importsys

importjson

frompyzabbiximportZabbixAPI

importdummy_inventory_api

ipaddr=sys.argv[1]

hostinfo_json=dummy_inventory_api.getinfo(ipaddr)

#hostinfo_jsonwillcontainaJSONstringsimilartothisone:

#{"hostip":"172.16.11.11",

#"hostname":"HostA",

www.it-ebooks.info

#"inventory":{

#"asset_tag":"12345678",

#"install_date":"31-11-2014",

#"installer_name":"SKL"

hostinv=json.loads(hostinfo_json)['inventory']

zbx=ZabbixAPI(http://127.0.0.1/zabbix/)

zbx.login("admin","zabbix")

hostinfo=zbx.host.get(output=['hostid'],filter={'ip':ipaddr})

hid=hostinfo[0]['hostid]

zbx_inventory={

'date_hw_install':hostinv['install_date'],

'installer_name':hostinv['installer_name'],

'asset_tag':'12345678'

#addotherfieldsyoumaybeinterestedin…

zbx.host.update(hostid=hid,inventory=zbx_inventory)

sys.exit()

Asyoucansee,thescriptisfairlystraightforwardandsimplistic,butitcanbeusedasastartingpointforyourowninventory-updatingscripts.Themainthingthatyouneedtotakecareofistofigureouthowtogetyourinventorydatafromyourassetdatabase.YoumightneedtoconnecttoaRESTAPI,orgetanXMLdocumentviaawebservice,orevenperformsomequeriesviaODBC.WhatmattersisthatyouendupwithaPythondictionaryorlistcontainingallthatyouneedtoupdatetherelevanthostinZabbix.

ThesecondpartofthescriptfirstofallshowsyouhowtoconnecttotheZabbixAPIusingtheZabbixAPIconstructor.Itthenproceedswiththeloginmethod,whereyou’llneedtoprovidethecredentialsyouconfiguredearlier.

Allgetmethodsacceptafilterparameterthatyoucanusetoretrieveasingleobjectoralistofobjectsthatsatisfycertainconditions.Inthiscase,weusedittogetthehostidofthehostthatisassociatedwithaspecificIPaddress.

Payattentiontothenextlineasthevaluereturnedbyallgetmethodsisalwaysalist,evenifitcontainsonlyoneelement.That’swhyweneedtoreferencethefirstelementofhostinfo,element0,beforereferencingtheinventorydictionarykey.

Weonlyshowedthreeinventoryfieldshere,buttherearemanymoreavailableinZabbix,soitmaybeagoodideatobuildadictionarywithallZabbixinventoryfieldsaskeysandtheretrievedvaluesasvalues.

Nowthatwehavethehostidandtheinventoryinformationatourdisposal,wecanproceedwiththeactualinventoryupdate.Theupdatemethodisfairlystraightforward:youspecifythehostidofthehostyouwanttoupdateandthenewvaluesforthefieldsthatyouneedtoupdate.

Andthat’sit,withascriptlikethisconfiguredasaremotecommandforadiscoveryaction,youcankeepyourZabbixinventorydatainsyncwithwhateverassetmanagementsystemyoumayhave.

www.it-ebooks.info

Asyoumighthaverealized,hostdiscoverycanbequiteacomplexmatterbecauseofthesheernumberofvariablesyouneedtotakecareof,andbecauseit’snotalwayseasy,inareal-worldnetwork,toidentifyaclearlogicforhostcreation,templateassignment,andothermonitoringparameters,basedondiscoverydata.

Low-leveldiscovery,bycontrast,ismuchmoresimple,givenitspowertodynamicallycreatespecificitemsasahost’savailableresourcesarediscovered.So,let’susetheremainingpagesofthischaptertoexploreafewaspectsofthisextremelyusefulfeature.

www.it-ebooks.info

Low-leveldiscoveryAnextremelyusefulandimportantfeatureofZabbixtemplatesistheirabilitytosupportspecialkindsofitemscalledlow-leveldiscoveryrules.Onceappliedtoactualhosts,theseruleswillquerythehostforwhateverkindofresourcestheyareconfiguredtolookfor:filesystems,networkinterfaces,SNMPOIDs,andmore.Foreveryresourcefound,theserverwilldynamicallycreateitems,triggers,andgraphsaccordingtospecialentityprototypesconnectedtothediscoveryrules.

Thegreatadvantageoflow-leveldiscoveryrulesisthattheytakecareofthemorevariablepartsofamonitoredhost,suchasthetypeandnumberofnetworkinterfaces,inadynamicandgeneralway.Thismeansthat,insteadofmanuallycreatingspecificitemsandtriggersofeveryhost’snetworkinterfacesorfilesystems,orcreatinghugetemplateswithanypossiblekindofitemforaparticularoperatingsystemandkeepingmostoftheseitemsdisabled,youcanhaveareasonablenumberofgeneraltemplatesthatwilladaptthemselvestothespecificsofanygivenhostbycreatingontheflyanyentityrequired,basedondiscoveredresourcesandpreviouslyconfiguredprototypes.

Outofthebox,Zabbixsupportsfourdiscoveryrules:

NetworkinterfacesFilesystems’typesSNMPOIDsCPUsandCPUcores(asofversion2.4)

Asdiscoveryrulesareeffectivelyspecialkindsofitems,youcancreateyourownrules,providedyouunderstandtheirpeculiaritycomparedtoregularitems.

Youneedtocreateandmanagelow-leveldiscoveryrulesintheDiscoveryrulessectionofatemplateconfigurationandnotintheusualItemssection,evenifthediscoveryrulesendupcreatingsomekindofitems.Themaindifferencebetweendiscoveredandregularitemsisthat,whereasaregularitemusuallyreturnsasinglevalue,adiscoveryitemalwaysreturnsalist,expressedinJSON,ofmacrovaluepairs.Thislistrepresentsalltheresourcesfoundbythediscoveryitems,togetherwithameanstoreferencethem.

ThefollowingtableshowsZabbix’ssupporteddiscoveryitemsandtheirreturnvalues,togetherwithageneralizationthatshouldgiveyouanideaofhowtocreateyourownrules:

Discoveryitemkey Itemtype Returnvalues

vfs.fs.discovery Zabbixagent

{"data":[

{"{#FSNAME}":<path>","{#FSTYPE}":"<fstype>"},

{"data":[

{"{#IFNAME}":"<name>"},

www.it-ebooks.info

net.if.discovery Zabbixagent {"{#IFNAME}":"<name>"},

snmp.discovery SNMP(v1,v2,orv3)agent

{"data":[

{"{#SNMPINDEX}":"<idx>","{#SNMPVALUE}":"<value>},

system.cpu.discovery Zabbixagent

{"data":[

{""{#CPU.NUMBER}":"<idx>","{#CPU.STATUS}":"<value>},

{"{#CPU.NUMBER}":"<idx>","{#CPU.STATUS}":"<value>},

custom.discovery Any

{"data":[

{"{#CUSTOM1}":"<value>","{#CUSTOM2}":"<value>"},

TipJustaswithallSNMPitems,theitemkeyisnotreallyimportantaslongasitisunique.It’stheSNMPOIDvaluethatyouaskanagentforthatmakesthedifference:youcancreatedifferentSNMPdiscoveryrulesthatlookfordifferentkindsofresourcesbychangingtheitemkeyandlookingfordifferentOIDvalues.Thecustomdiscoveryexampleisevenmoreabstractasitwilldependontheactualitemtype.

Asyoucansee,adiscoveryitemalwaysreturnsalistofvalues,buttheactualcontentsofthelistchange,dependingonwhatresourcesyouarelookingfor.Inthecaseofafilesystem,thereturnedlistwillcontainvalueslike{#FSNAME}:"/usr",{#FSTYPE}:"btrfs",andsoonforeverydiscoveredfilesystem.Ontheotherhand,anetworkdiscoveryrulewillreturnalistofthenamesofthediscoverednetworkinterfaces.ThisisthecaseforthedefaultSNMPnetworkinterfacestemplate.Let’sseeindetailhowitworks.

Thetemplatehasadiscoveryrulecallednetworkinterfaces.Itlooksjustlikearegularitemasithasaname,atype,anupdateinterval,andakey.It’sanSNMPtype,soitalsohasanSNMPOID,IF-MIB::ifDescr.Thisisadiscoveryrule,soinsteadofasinglevalue,itwillreturnalistofalltheOIDsthatarepartoftheIF-MIB::ifDescrsubtreeforthatparticulardevice.ThismeansthatitwillreturntheOIDanditsvalueforallthenetworkinterfacespresentonthedevice.Everytimethediscoveryruleisexecutedonahost(basedontheupdateinterval,justlikeanyotheritem),itwillreturnalistofallinterfacesthatareavailableatthatparticularmoment.Ifthedevicehadfournetworkinterfaces,itcouldreturnsomethingsimilartothis:

{"data":[

{"{#SNMPINDEX}":"1",

"{#SNMPVALUE}":"FastEthernet0/0"},

www.it-ebooks.info

Thediscoveryrulewillthenproceedtoapplythelisttotheitemandtriggerprototypesithasconfigured,asfollows:

TakingtheIncomingtrafficoninterface{#SNMPVALUE}itemprototypeasanexample,youcanseehowitallcomestogether:

The{#SNMPVALUE}macroisusedintheitem’skeyand,therefore,intheitem’snameaswell(lookatthe$1macrothatreferencesthefirstargumentoftheitem’skey).

www.it-ebooks.info

Ontheotherhand,the{#SNMPINDEX}macrowillbeusedbyZabbixtoactuallygettheincomingtrafficvalueforthatspecificinterfaceasitshouldbeclearbynowifyouobservethevalueintheSNMPOIDfield.

Whenconfiguringatemplate’sdiscoveryrules,youdon’tneedtocareabouttheactualvaluesreturnedintheirlists,northelists’length.Theonlythingyouhavetoknowisthenameofthemacrosthatyoucanreferenceinyourprototypes.Thesearetobereferencedinthesecondhalfofthelow-leveldiscoverymechanism,objectprototypes.Youcreatethemasregulartemplateentities,makingsureyouusethediscoveryitemmacroswhereneeded,andZabbixwilltakecareoftherestforyou,creatingforeachitemprototypeasmanyitemsasthereareelementsinthelistreturnedbythediscoveryrule,foreachtriggerprototypeasmanytriggersasthereareelementsinthelistreturned,andsoon.

So,whenyouapplythetemplatetoahost,itwillcreateitems,triggers,andgraphsbasedontheresourcesdiscoveredbythediscoveryitemsandconfiguredaccordingtothediscoveryprototypes.

Customdiscoveryrules,fromthispointofview,workexactlyinthesamewayascustomitems,whetheryoudecidetouseagent-sidescripts(therebyusingacustomzabbix.agentitemkey),externalscripts,databasequeries,oranythingelse.Theonlythingsyouhavetomakesureofisthatyourcustomitemsreturnkeys/valuesthatfollowtheJSONsyntax,asshownintheprecedingtable,andthatyoureferenceyourcustommacrosintheentitiesprototypesthatyouwillcreate.

Let’sseeanexampleofacustomdiscoveryruleusingagainNmapanditsoutputtodynamicallycreatesomeitemsforahost,representingtheopenportithas,andthekindofservicesthatarelistening.WhywouldyouwanttouseNmapandaportscan?Thedeviceyouneedtomonitormaybedoesn’tsupporttheZabbixagent,soifyoujustaskfortheoutputofnetstat,youmightnotbeabletoinstalltheagentforadministrativereasons,oryoumighthavetomakesurethattheservicesarealsoavailablefromanothernetwork,socheckingthemfromafar,insteadofdirectlyonthehost,willenableyoutoalsoverifyyourfirewallrules,killingtwobirdswithonestone.

Eitherway,we’llcreateanexternalcheckitemperopenTCPport,configuredasacharacter-typeitem.Eachitemwillcontainthenameoftheservicethatwasfoundlistening,ifany,asreportedbyNmap’sservicediscoveryfacilities.

Startbycreatingthediscoveryruleasanexternalcheckthatwillcallaport-mappingscript,asfollows:

www.it-ebooks.info

Asyoucansee,thescriptwillreceivethehost’sIPastheonlyargument,anditwillrunonceanhourforeveryhostthathasthisdiscoveryruleconfiguredandisactive.

ThescriptitselfisverysimpleandisbasedonNMAP’sXMLoutputcoupledwiththeniftyxml2toolyoualreadyusedinChapter3,MonitoringYourNetworkServices,asfollows:

#!/bin/bash

IPADDR=$1

#storeportsasarray

PORTS=($(nmap-sV-oX-${IPADDR}|xml2|grepportid|cut-d'='-f2))

#countelementsofthearrayanduseascounterforlaterprocessing

COUNTER=${#PORTS[@]}

#openJSON

echo'{"data":['

#loopthroughportsandprintkey/value

forPORTin"${PORTS[@]}";do

COUNTER=$((COUNTER-1))

if[$COUNTER-ne0];then

echo"{\"{#PORTID}\":\"${PORT}\"}",

#it'sthelastelement.TohavevalidJSONWedon'taddatrailingcomma

echo"{\"{#PORTID}\":\"${PORT}\"}"

#closeJSON

echo]}

#exitwithcleanexitcode

Thelinestartingwithnmapistheheartofthescript.The–oXoptionenablesXMLoutput,whichismorestableandeasytomanagecomparedtothenormalone.Thedashafter–oXspecifiesstdoutastheoutputinsteadofaregularfile,sowecanpipetheresulttoxml2andthentakeonlythelinesthatcontainportid,thatis,theopenportnumbersforthathost.

www.it-ebooks.info

Asaresult,thescriptjustoutputsasimpleJSONobject.Here’sanexampleofwhatthediscoveryrulewillget,asshownfromthecommandline:

./port_map.sh'127.0.0.1'

{"data":[

{"{#PORTID}":"22"},

{"{#PORTID}":"25"},

{"{#PORTID}":"80"},

{"{#PORTID}":"631"},

{"{#PORTID}":"3306"}

It’snowtimetodefinetheitemandtriggerprototypes,basedontheopenportthatyoufound.We’llshowhereanexampleofanitemprototypethatwillreturnthenameandversionofthedaemonlisteningontheport,asreturned,onceagain,byNmap:

Theexternalcheckwillcallascriptthatisevensimplerthanthepreviousone,asfollows:

#!/bin/bash

IPADDR=$1

PORT=$2

nmap-sV-oX--p${PORT}${IPADDR}|xml2|grep'port/service/@\

(product\|version\|extrainfo\)'

ComparedtothepreviousNmapcommand,weaddeda–sVoptiontomakeNMAPrunaseriesofprobesinordertofindoutwhatserviceisrunningbehindthatopenportanda–poptiontospecifyasingleporttoscan.

Theoutputwaskeptsimpleonpurposetoshowyouanexampleofxml2’soutput.Youcan,ofcourse,sliceitanddiceittosuityourownneeds:

./port_service.sh127.0.0.180

/nmaprun/host/ports/port/service/@product=Apachehttpd

/nmaprun/host/ports/port/service/@version=2.2.15

/nmaprun/host/ports/port/service/@extrainfo=(CentOS)

NoteTheamountofinformationNmapwillbeabletogetfromanetworkservicedependsvery

www.it-ebooks.info

muchonhowmuchandonwhatkindofdatatheserviceisconfiguredtoexpose.Thismightdependonbuilt-inparametersorsecurityconsiderationsonthepartoftheserviceowner.Comparedtothepreviousexample,yourmileagecanvary.

Thisiswhatwillappearasthevalueoftheitemoncethediscoveryruleisactivated.

www.it-ebooks.info

SummaryInthischapter,youlearnedhowtouseZabbix’sdiscoveryfacilitiestoautomateitsconfigurationasmuchaspossible.Itshouldalsobecleartoyouwhyit’simportanttominimizethedifferencebetweenwhatisconfiguredinZabbixandwhatisactuallyoutthereonthewire.Keepingtrackofeverythingthatcanappearordisappearonabusynetworkcanbeafulltimejobandonethatisbettersuitedtoautomatedmonitoringfacilitieslikethisone.Younowhavealltheskillsneededtoactuallydoit,andyouarereadytoapplytheminyourreal-worldenvironment.

Inthenextchapter,we’llwrapthingsupbyshowingyouhowtoleverageZabbix’spresentationpowertocreateandmanagegraphs,dynamicmaps,andscreens.

www.it-ebooks.info

Chapter5.VisualizingYourTopologywithMapsandGraphsAsyouprobablyalreadyknow,Zabbix’sapproachtomonitoringisbasedonseparatingdatagatheredfromtriggerlogicandeventlogging.Ontheonehand,thismeansthatyouareabletoreferenceanymeasurement,presentandpast,inyourtriggers,makingthemallthemorepowerful.Ontheotherhand,italsomeansthatyouhavedirectaccesstoallyourmeasurementhistoryforallyouritems.

Whilesortingthroughallofyourhistoricaldatatolookforaspecificvaluecancertainlybeuseful,therealadvantagehereistoleverageZabbix’sgraphingandmappingfunctionalitiestoaggregateandvisualizedatainmeaningfulways.

Inthischapter,you’llseehowtocreatecomplexgraphsfromyouritems’numericalvalues,howtoautomaticallydrawmapsthatreflectthecurrentstatusofyournetwork,andhowtobringitalltogetherusingscreensasatooltocustomizemonitoringdatapresentation.

www.it-ebooks.info

CreatingcustomgraphsBasicgraphicaldatarepresentationcomesforfreeforanyitemthathasanumericdatatype.YoujustneedtogotoMonitoring|LatestData,selectthehostyouareinterestedin,findtherelevantitem,andclickonGraphinthelastcolumnontheright-handside.You’llgetalinegraphwithatimesliderthatyoucanusetochangethetimeframeofthegraphitself;widenittocoveralongeramountoftime,orshortenittofocusonaspecificpointintime.

SinceZabbix2.4,youcanalsocomparedifferentitemsontheflywithadhocgraphs.Theseareadirectextensionofsimplegraphs:fromMonitoring|LatestData,youjustneedtomarkthecheckboxontheleft-handsideofeveryitemthatyouwanttographandselectDisplaystackedgraphorDisplaygraphfromthedrop-downmenuatthebottomofthepage,asfollows:

Theresultisprettymuchtheoneyouexpect.Youalsodon’thavetoworrytoomuchaboutchoosingbetweenanormalgraphandastackedgraphasyou’llbeabletoswitchbetweenthetwofromthegraphitself,asfollows:

Thesequick,adhocgraphscanreallycovermostofyourvisualizationneeds,especiallyforvaluesthatyoudon’tconsultthatoftenorifyouneedtocompareitemsthatyou

www.it-ebooks.info

normallydon’thaveto,aspartofanewanalysisortoinvestigateanewclassofproblems.

Ontheotherhand,ifyouneedtocomparethesametypesofitemsoverandover,andfordifferenthosts,you’llneedawaytosaveyourselectionssothatyouareabletoaccessyouraggregatedgraphswithouthavingtospecifyeverytimewhatitemsneedtobegraphed.Youcanachieveallthiswithcustomgraphs.

NoteIfyouliketovisualizeyourpercentiledatawithpiecharts,you’llalsoneedtocreatecustomgraphsasthey’recurrentlytheonlywaytocreatepiechartsinZabbix.

Customgraphscanbecreatedaspartofahost,orbetteryetaspartofatemplate,oralow-leveldiscoveryrule,sothatanyhostinheritingthetemplateordiscoveryrulewillautomaticallyalsoinheritthecustomgraph.

Tocreateone,youneedtogotoConfiguration|Templates,choosethetemplateyouwanttoputyourgraphinto,selectGraphs,andclickonCreategraph.Thiswillbringyoutothegraphcreationform.Forconvenience,thefollowingexamplewillshowyousomeitemsalreadyaddedtotheitemlistandsomeotheroptionsalreadyselectedinsteadofanemptyform,butyou’lleasilybeabletoaddyourownitemsbyfollowingtheaddlinkatthebottomoftheitemlist,asfollows:

Asyoucansee,thereareafewoptionsworthnoting.Firstofall,youcanselectthegraphtypebetweenNormal,Stacked,Pie,andExploded(thatis,apiechartwithallslicesseparatedinsteadofclosetogether).Next,ifyouselecttheShowtriggerscheckbox,thegraphwillincludeahorizontallineforeverytriggerthathasanyoftheitemspresentin

www.it-ebooks.info

thegraph’sitemlistinitsexpression.Youdon’thavetospecifythetriggerorfindthemmanually;Zabbixwilltakecareoffindingallrelevanttriggersandshowthemonthegraph.

Youcanalsospecifytherangeofyaxisvalueseitherasfixedvaluesorcalculatedbasedonthedatayouhave.You’llnormallywanttosetthemascalculatedasthisoptionwillusuallyshowtheclearestandbest-lookinggraphs,butsometimes,youmightwanttosetthemtoafixedvaluetohaveabetterunderstandingofhowthevalueschange,especiallyiftheyfluctuatealotbetweenverybigandverysmallvalues,andtheitemexpressesapercentilerange.

Movingtotheitemlist,youcanordertheitemsbydragginganddroppingthebluearrowsontheleft-handsideoftheitem’snameandchangetheircolorbyeitherspecifyinganRGBvalueorchoosingfromacolorpalette.

Thedrawstylecanbequiteusefulifyouwantaspecificitemtostandoutfromtherest.Therearequiteafewstylesavailableforanormalgraph,whilethisoptionisnotavailableforstackedandpiecharts.

TheFunctiondrop-downmenuenablesyoutochoosehowtheitemshouldbegraphedforeverytickinthexaxis:youcanchoosebetweentheminimumvalue,themaximumone,andtheaverage.Keepinmindthatthex-axistickdensitywillchangedynamicallywiththetimescaleofthegraph(youcanselectdifferenttimeframeswhilelookingatagraph;youdon’thavetospecifytheminadvance):fortimeframesuptoanhour,itwillshoweverysamplecollected,dependingontheitems’samplefrequency;forlargertimeframes,you’llhavex-axisticksproportionaltothetimeframeselected,whichisafewminutesiftheglobaltimeframeisafewhours,todaysorweeksifyouselectmonths’oryears’worthofmonitoringdata.Foreverytick,Zabbixwillusethefunctionyouselectedheretoplottheitemvalueeitherbyselectingthemaximum,theminimum,ortheaveragevalueforthattimetick.

Finally,youcanchoosewhethertheyaxisforanitemwillbeshownontheleft-handsideortheright-handside.Oneofthereasonstoseparatedifferentitemsondifferenty-axissidesisthatmaybeyouareplottingonthesamegraphitemsthathaveabsolutevaluestogetherwithitemsthatexpressapercentilevalue.Inthiscase,itmakessensetoshowtheabsolutescaleononesideandthepercentileoneontheothersideofthegraph.

Anotherreasonmightbethatyouareplottingtogetheritemsthatwillshow,onaverage,verybigorverysmallvalues,andyoucanpredictaheadoftimetheonesthatwillgravitatetowardsthebottomofthescale,andtheonesthatwillmakethescalegoupwithbigvalues.Inthatcase,youmightwanttoseparatethetwo;otherwise,theitemswithbigvalueswillmaketheotherslookveryflatandnotveryinformativeonthechart.Thisisthecaseillustratedintheprecedinggraph:wepredictedthatthetotalnumberofquerieswouldbemuchbigger(bydefinition)comparedtoalltheothers,sowemoveditsyaxistotheright-handside.Here’stheresultofthegraphwecreated:

www.it-ebooks.info

Whatwehaven’tshownhere,butyoucaneasilyimagine,isthataswithalmosteverythinginZabbix,youarenotlimitedtographingitemsfromthesamehost:youcanjustaseasilygraphthesameitemfromdifferenthosts,orevendifferentitemsfromdifferenthosts.Youmightbeinterested,forexample,intrackingnetworktrafficfromabunchofdifferentroutersandlookingathowthistrafficchangesintime,whichmachinesarethebusiestandwhen,whichonesarenotasbusyasyouexpectedcomparedtotheoveralltrafficyouhave,andsoon.Todothat,youcaneasilycreateagraphfollowingtheguidelinesabove,onlyselectingtherelevantnetworkinterfacesinboundandoutbounditemsfromthedifferentappliancesandputtingthemallonthesameitemlist.

YoucanuseZabbix’scustomgraphcreationfacilitiestoexploreyourdatainverymeaningfulwaysthatcanbehardtoachieveotherwise:don’tbefooledbythefactthatit’sallmainlytime-based(youcan’tputcustomvaluesonthexaxis).You’llsoonfindthattheabilitytocorrelatedifferentitemsfromdifferentsourcesisaverypowerfultoolforbothtroubleshootingandcapacityplanning.

AnotherpowerfultoolisZabbix’smappingfacility.We’llexploreafewinterestingaspectsofmapcreationandmaintenanceinthefollowingsection.

www.it-ebooks.info

Maps–aquicksetupforalargetopologyCreatingcomplexmapsisthekindofjobthatcantakealotoftime.Whiledoingapracticalexample,ifyouwouldliketodesignamapof20-30elements,itiseasytospendupto2hoursevenifyoualreadyknowthejob.

Tomanuallyproduceamap,youneedto:

AddalltheitemsonthemapMovetheitemsarounduntilyouseeanice-lookingdisposition

Everytimeyouneedtoaddinamaponehost,youneedtorepeatmanytimesthesamestepsasaforementioned,whichwillbecomeaboringandcomplextask.Currently,therearemanyopen-featurerequeststhatcanfacilitatethiskindoftask;unfortunately,theyhavebeenopenforalongtime,evenyears.

Theissuesyoucanfaceare:

Youcan’tmovemultipleelementsatthesametime,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-161Youcan’taddhostsinabulkway,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-163Youcan’tcloneanyexistingmapelement,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-51Whenyouareusingicons,youcan’tselectthemautomatically,soyouneedtochecktheirsizeandseewhethertheyfitonyourmap,somethingthatcanbefoundathttps://support.zabbix.com/browse/ZBXNEXT-1608

Forallthoseissues,weneedtofindadifferentwaytoautomatethislongandslowprocess.Clearly,thisisthekindoftaskthatneedstobeautomatedasmuchaspossible.

www.it-ebooks.info

Maps–automatingtheDOTcreationWhatismissinghereissomethingthatcanprocessourinformationandproduceasoutputsomethingusablebyZabbix.Toautomatethistask,thereisonelibrarythatcanhelpus—NetworkX—whichisavailableathttp://networkx.github.io/.

NetworkXisaPythonsoftwarelibrarytailor-madeforthecreation,manipulation,andstudyofdynamicnetworkstructures.

Inthisexample,weassumethatyou’reusingCiscoPrime,whichisavendor-specifictooltoexportadiscoveredtopology.

Anyway,thisconceptisstillvalidasherewearegoingtouseanexportfileobtained,whichisinCSV.ThiskindofCSVcanbeobtainedasanexportfrommanyothervendors’softwareandcanbeeasilyproducedfromanythird-partysoftware.

Thefilethatwearegoingtoparseisinthefollowingform:

IPaddress,Systemname,SysObjectID,Foundbymodules,Neighbors,Status

Asyoucansee,itcontainstheIPaddressofthedevicediscovered,thesystemname,theOIDofthesystem,themodulethatfoundthedevice,alistofalltheneighborsthatareconnectedtoit,anditendswiththestatus.

Thefollowingisanexampleofthelinethatweareexpectingtosee:

10.12.50.1,main.example.com,.1.3.6.1.4.1.9.1.896,System,"10.12.2.1,

10.12.2.2,10.12.3.1,10.12.4.1,10.12.5.1",Reachable

Wearemostlyinterestedinthefollowingfields:

IPaddressSystemnameSysObjectIDNeighbors

Then,whatwecandoiswritesomePythonlinesthatcanreadthisfile,identifyalltherequiredinformation,andwriteintheoutputaDOTfile.

Here,IamgoingtospendafewwordsabouttheDOTnotation,performinganexampleinordertoclarifyhowthisnotationisdone.

Firstofall,IwouldliketoexplainwhywearegoingtohaveaGraphvizDOTfile.

TheGraphvizDOTfileisreallyeasytoread,maintain,andupdate,andnevertheless,itcanbestoredinaCVSorSVN.

Somethingthatisreallyimportanttohaveisafilethatcanbequicklyusedtospotallthedifferencesbetweenversionsandiseasytomaintain.Also,weareconsideringusingitasitisastandardlanguageandagoodstartingpoint,onwhichwecantransformallouracquireddatafromallthedifferentversionsofexport.

Indeed,someothervendor-specificsoftwarecanexportthesamedatabutinadifferentform,soitisimportanttonormalizeallourdatainacommonlanguage.

www.it-ebooks.info

ThiscommonlanguagefilewillbethefiletousetopopulateourZabbixmap.

Thissection,asyouprobablyalreadyhaveunderstood,willbealargeusageoftheGraphviz’spackages.

TheeasiestwaytoinstallandmaintainGraphvizonRedHatEnterpriseLinuxistousethededicatedyumrepository.Tosetupyum,firstofall,youneedtodownloadthegraphviz-rhel.repofileandsaveit(asroot)in/etc/yum.repos.d/,asfollows:

#cd/etc/yum.repos.d

#wgethttp://www.graphviz.org/graphviz-rhel.repo

--2014-11-2702:52:17--http://www.graphviz.org/graphviz-rhel.repo

Resolvingwww.graphviz.org…204.178.9.49

Connectingtowww.graphviz.org|204.178.9.49|:80…connected.

HTTPrequestsent,awaitingresponse…200OK

Length:1138(1.1K)[text/plain]

Savingto:"graphviz-rhel.repo"

100%[======================================>]1,138--.-K/sin0s

2014-11-2702:52:17(134MB/s)-"graphviz-rhel.repo"saved[1138/1138]

#ls-lagraphviz-rhel.repo

-rw-r--r--.1rootroot1138Feb162012graphviz-rhel.repo

Then,youcanfinallylistalltheGraphvizpackagesasroot:

yumlistavailable'graphviz*'

Installthem,asfollows:

yuminstall'graphviz*'

Nowthatwe’veclarifiedthereasonwhywe’redoingthosesteps,itisimportanttowalkthroughtheDOTlanguage.TheDOTlanguageisalanguagemadetorepresentobjectsconnectedbetweeneachother.

Whileperformingapracticalexample,ifwewanttodefinetwoconnectednodeswiththeGraphvizDOTlanguage,wecandoasfollows:

graph{

Thisisaveryeasy-to-understandlanguage;wearenowrepresentingtwonodesconnectedtoeachother.

Toseethegraphicalresult,wecanuseasimplePythonprogramxdot.pyavailablefordownloadhere:

https://github.com/jrfonseca/xdot.py

Allyouhavetodoisdownloadtheprogram,writeafilewiththeGraphvizDOTcontentthatweshowedpreviously,andthenruntheprogram,asfollows:

xdot.pyexample.dot

www.it-ebooks.info

TheresultistheDOTexpressedtopologyvisualized,asfollows:

Usingthesamegrammar,wecandefinethreenodesconnected,asfollows:

graph{

A—B—C

Usingthesamexdot.pyusedpreviously,theresultisthefollowing:

Writingacoupleoflinesmore,wecanevenavoidusinglongnamesusingthefollowinggrammar:

graph{

//Wecancreatealiasestoavoidtouseverylongnamesonthedependency

definition

Andrea[hostname="andrea.dalle.vacche.example.com"]

Stefano[hostname="stefano.kewan.lee.example.com"]

router[label="Ournetworkrouter"zbximage="router"]

//nowit'stimetodefineconnectionsbetweenthenodes

//Thisnotationallowsformultipleedgesfrom"router"inonego

router—{AndreaStefano}

www.it-ebooks.info

Andtheresultisshownhere:

Foradetaileddocumentationofthisgrammar,pleaserefertotheofficialdocumentationavailableathttp://www.graphviz.org/content/dot-language.

Untilnow,we’vecoveredallthatisneededtoknowforoursmallapplication.

Now,wecancomebacktoourCSVfileweextractedfromCiscoPrime.

HereistheCSVofaverysimplenetwork,butitcanbeappliedonverycomplexnetworktopologies,aswell:

[root@localhostgraphs]#catmy_export.csv

IPAddress,SystemName,SysObjectID,FoundByModules,Neighbors,Status

10.12.20.1,main.example.com,.1.3.6.1.4.1.9.1.896,System,"10.12.2.1,

10.12.2.2,10.12.3.1,10.12.4.1,10.12.5.1",Reachable

10.12.2.1,cluster1.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.2.2,

192.168.99.1",Reachable

10.12.1.1,london.example.com,.1.3.6.1.4.1.9.1.503,System,"",Reachable

10.12.2.2,cluster2.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.2.1,

192.168.99.1",Reachable

10.12.3.1,switch1.example.com,.1.3.6.1.4.1.9.1.503,System,"192.168.99.1",Re

achable

10.12.4.1,4.example.com,.1.3.6.1.4.1.9.1.502,System,"192.168.99.1,

10.12.4.42,10.12.4.47,10.12.4.48,10.12.4.49",Reachable

10.12.4.45,4d.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable

10.12.4.46,4e.example.com,.1.3.6.1.4.1.9.1.502,System,"10.12.4.1",Reachable

10.12.4.47,4f.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable

10.12.4.48,4g.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.4.1",Reachable

10.12.5.45,10.12.5.43,10.12.5.44,10.12.5.46,10.12.5.47,10.12.5.48,

10.12.6.1",Reachable

10.12.5.44,5c.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable

10.12.5.48,5g.example.com,.1.3.6.1.4.1.9.1.503,System,"10.12.5.1",Reachable

10.12.5.155,5i.example.com,.1.3.6.1.4.1.9.1.634,System,"10.12.5.1",Reachabl

10.12.6.46,10.12.6.47,,10.12.5.1",Reachable

www.it-ebooks.info

Fromthisfile,weseethatalltherelationsbetweenneighborsarealreadycontainedintheCSV,andthatweonlyneedtoconvertthemintoDOTnotationusingthenodenotation.

Here,wecanstartcodingafewPythonlinestoproduceourdesiredoutput:

#FirstofallweneedtoimportcsvandNetworkx

importcsv

importnetworkxasnx

#Thenweneedtodefinewhoisourzabbixserverandsomeotherdetailto

properlyproducetheDOTfile

zabbix_service_ipaddr="192.168.1.100"

main_loop_ipaddr="10.12.20.1"

main_vlan_ipaddr="149.148.56.1"

#Nowwecanfinallycreateourgraph

G=nx.Graph()

#wecanopenourCSVfile

csv_reader=csv.DictReader(open('my_export.csv'),\

delimiter=",",\

fieldnames=("ipaddress","hostname","oid","dontcare","neighbors"))

#Skiptheheader

csv_reader.next()

forrowincsv_reader:

neighbor_list=row["neighbors"].split(",")

forneighborinneighbor_list:

#Removespaces

neighbor=neighbor.lstrip()

#Addneighbors,andherewe'vedecidedtoignoreisolatednodes

ifneighbor!="":

G.add_edge(row["ipaddress"],neighbor)

#Addadditionalinformationtonodesoredgeshere

G.node[row["ipaddress"]]["hostname"]=row["hostname"]

#CiscoPrimedoesn'texportallIPaddressesofadevice

#butonlythefirstforeachnetwork,Herewemergehostswith

#multipleIPaddresses

mapping={main_vlan_ipaddr:main_loop_ipaddr}

G=nx.relabel_nodes(G,mapping)

#Removeclusterconnectionnotneededinourmap

G.remove_edge("10.12.2.1","10.12.2.2")

#AddingconnectionbetweenZabbixserverandmainswitch

G.add_edge(zabbix_service_ipaddr,main_loop_ipaddr)

main_neigh_list=G.neighbors(main_loop_ipaddr)

#finallywriteoutourfile

nx.draw_graphviz(G)

nx.write_dot(G,"/tmp/total.dot")

Now,ifyourunthissmallsoftwareagainsttheCSVfilewehaveshownbeforeyouseeourDOTfilegeneratedon/tmp/total.dot.Now,itisinterestingtoseehowourDOTfile

www.it-ebooks.info

isrepresentedonXDot.Here,inthenextdiagram,weseetherepresentationofourDOTfile:

Now,allthatwehavetodoisproducethemapstartingfromtheDOTfilewejustgenerated.

www.it-ebooks.info

DraftingZabbixmapsfromDOTHavingarrivedatthispoint,wehaveourGraphvizDOTfilethatiswaitingtobeused.Asyoucanseefromthepreviousimage,thankstoGraphviz,wealreadyhaveaready-to-goimagetouse.Then,allweneedtodois:

1. ReadouttheDOTfile.2. GeneratethetopologyusingGraphviz.3. Acquireallthecoordinatesfromourtopologygenerated.4. UsepyzabbixtoconnecttoourZabbixserver.5. Generateourtopologyinafullyautomatedway.

It’snowtimetowritesomelinesofPython;thefollowingexampleissimilartosomethingpresentedbyVolkerFröhlich.Anyway,thecodeherehasbeenchangedandfixed(itdidnotworkwellwithZabbix2.4).

Asthefirstthing,weneedtoimporttheZabbixApiandnetworkXlibraries:

importnetworkxasnx

frompyzabbiximportZabbixAPI

Then,wecandefinetheGraphvizDOTfiletouseasasource;agoodexampleistheonewejustgenerated:

dot_file="/tmp/total.dot"

Inthenextfewlines,wedefineourusername,password,mapdimension,andrelativemapname:

username="Admin"

password="zabbix"

width=800

height=600

mapname="my_network"

Whatfollowsisastaticmaptodefinetheelementtype:

ELEMENT_TYPE_HOST=0

ELEMENT_TYPE_MAP=1

ELEMENT_TYPE_TRIGGER=2

ELEMENT_TYPE_HOSTGROUP=3

ELEMENT_TYPE_IMAGE=4

ADVANCED_LABELS=1

LABEL_TYPE_LABEL=0

Then,wecandefinetheiconstouseandtherelativecolorcode:

icons={

"router":23,

"cloud":26,

"desktop":27,

"laptop":28,

"server":29,

"sat":30,

www.it-ebooks.info

"tux":31,

"default":40,

colors={

"purple":"FF00FF",

"green":"00FF00",

"default":"00FF00",

Now,wedefinesomefunctionsthatwecanreuse.Thefirstoneistomanagethelogin,andthesecondoneistodefineahostlookup,asfollows:

defapi_connect():

zapi=ZabbixAPI("http://127.0.0.1/zabbix/")

zapi.login(username,password)

returnzapi

defhost_lookup(hostname):

hostid=zapi.host.get({"filter":{"host":hostname}})

ifhostid:

returnstr(hostid[0]['hostid'])

Thenextthingtodo,isreadourDOTfileandstartconvertingitintoagraph:

G=nx.read_dot(dot_file)

Then,wecanfinallyopenourgraph,asfollows:

pos=nx.graphviz_layout(G)

NoteHere,youcanselectyourpreferredalgorithm.Graphvizsupportsmanydifferentkindsoflayout,andthenyoucanchangethelookandfeelofyourmapasyouprefer.FormoreinformationaboutGraphviz,pleasechecktheofficialdocumentationavailableathttp://www.graphviz.org/.

Then,asthegraphisalreadygenerated,thenextthingtodoisfindthemaximumcoordinatesofthelayout.Thiswillenableustoscalebetterourpredefinedmapoutputsize.

positionlist=list(pos.values())

maxpos=map(max,zip(*positionlist))

forhost,coordinatesinpos.iteritems():

pos[host]=[int(coordinates[0]*width/maxpos[0]*0.95-

coordinates[0]*0.1),int((height-

coordinates[1]*height/maxpos[1])*0.95+coordinates[1]*0.1)]

nx.set_node_attributes(G,'coordinates',pos)

NoteGraphvizandZabbixusetwodifferentdataorigins:Graphvizstartsfromthebottom-leftcorner,andZabbixworksstartingfromthetop-leftcorner.

Then,weneedtoretrievetheselementidsastheyarerequiredforlinksandevenforthenodedatacoordinates,asfollows:

www.it-ebooks.info

selementids=dict(enumerate(G.nodes_iter(),start=1))

selementids=dict((v,k)fork,vinselementids.iteritems())

nx.set_node_attributes(G,'selementid',selementids)

Now,wedefinethemaponZabbix,thename,andtherelativemapsize:

map_params={

"name":mapname,

"label_type":0,

"width":width,

"height":height

element_params=[]

link_params=[]

Finally,wecanconnecttoourZabbixserver:

zapi=api_connect()

Then,prepareallthenodeinformationandthecoordinatesandthensettheicontouse,asfollows:

fornode,datainG.nodes_iter(data=True):

#Genericpart

map_element={}

map_element.update({

"selementid":data['selementid'],

"x":data['coordinates'][0],

"y":data['coordinates'][1],

"use_iconmap":0,

Checkwhetherwehavethehostname,asfollows:

if"hostname"indata:

"elementtype":ELEMENT_TYPE_HOST,

"elementid":host_lookup(data['hostname'].strip('"')),

"iconid_off":icons['server'],

"elementtype":ELEMENT_TYPE_IMAGE,

"elementid":0,

Wesetlabelsforimages,asfollows:

if"label"indata:

"label":data['label'].strip('"')

if"zbximage"indata:

"iconid_off":icons[data['zbximage'].strip('"')],

www.it-ebooks.info

elif"hostname"notindataand"zbximage"notindata:

"iconid_off":icons['default'],

element_params.append(map_element)

Now,weneedtoscanalltheedgestocreatetheelementlinksbasedontheelementweidentified,asfollows:

nodenum=nx.get_node_attributes(G,'selementid')

fornodea,nodeb,datainG.edges_iter(data=True):

link={}

link.update({

"selementid1":nodenum[nodea],

"selementid2":nerodenum[nodeb],

if"color"indata:

color=colors[data['color'].strip('"')]

link.update({

"color":color

link.update({

"color":colors['default']

if"label"indata:

label=data['label'].strip('"')

link.update({

"label":label,

link_params.append(link)

#Jointhepreparedinformation

map_params["selements"]=element_params

map_params["links"]=link_params

Now,wehavepopulatedallmap_params,andnowweneedtocallZabbix’sAPIwiththisdata:

map=zapi.map.create(map_params)

Theprogramisnowcomplete,andwecanletitrun!Inareal-worldcase,thetimespenttodesignatopologyofmorethan2,500hostsisonly2–3seconds!

Wecantestthesoftwarehere,proposedagainsttheDOTfilewegeneratedbefore:

[root@localhost]#time./Generate_MyMap.py

real0m0.005s

user0m0.002s

sys0m0.003s

Asyoucansee,oursoftwareisreallyquick…butlet’scheckwhathasbeengenerated.In

www.it-ebooks.info

thenextscreenshot,youcanseethemapthatisgeneratedautomaticallyin0.005seconds:

www.it-ebooks.info

PuttingeverythingtogetherwithscreensUnlikeanyotherZabbixfeaturewedescribedinthischapter,screensdon’tactuallygiveyouneworimprovedinformationaboutyourmonitoreddata.PrettymuchanythingthatyoucandecidetoputonascreencanbefoundsomewhereelseinZabbix.

Frommapsandgraphs,totriggerstatusanditemdata,allofthisandmorecanbeeasilyfoundbyexploringtheMonitoringtabofthewebfrontend.

ButthepointofgatheringexistingdataonaZabbixscreenispreciselythatyoubringtogetherrelateddata,ordifferentviewsofthesamedatasothatyoudon’thavetolookforitaroundthefrontend,andsothatyoucanhaveagoodoverviewofthestatusofyoursystemsandseeataglancewhetherthereareanyproblemswithinyourinfrastructure.

Whenyoucreateascreen(Configuration|Screens|Createscreen),yougiveitanameandastartingnumberofrowsandcolumns.Don’tworrytoomuchabouthowmanyrowsandcolumnsyouassigntoascreenasyouwillbeabletochangethemduringscreenconfiguration.

Onceyouhavethescreencreated,youcangoaheadandconfigureitbyselectingitsnameinConfiguration|Screens.

Ascreenisbasicallyatablewithrowsandcolumnsthatidentifiescells.Everycellcancontaindifferenttypesofdata:

Celltype Description

Actionlog ThisshowsalogofthelatestactionsexecutedbyZabbix.Youcanconfigurehowmanyactionsyouwanttoseeinthecell.

Clock Thisshowsananalogclockwiththecurrenttime.

Dataoverview Thisshowsthelatestitemdataforaspecificgroupofhosts.

Graph Thisshowsanexistingcustomgraph.

Graphprototype Thisshowsacustomgraphcreatedfromalow-leveldiscoveryruleprototype.

Historyofevents

Thisshowsalogofthelatestevents(thesedon’tnecessarilyleadtoactions).Youcanconfigurehowmanyeventsyouwanttoseeinthecell.

Hostgroupissues Thisshowsthecurrentissuesforaspecifichostgroup.

Hostissues Thisshowsthecurrentissuesforaspecifichost.

Host’sinfo Thisshowsasummaryofhostavailabilityforaspecificgroup,suchastheoneyoufindinMonitoring|Overview.

Map Thisshowsanexistingmap.

www.it-ebooks.info

PlaintextThisshowstheplaintexthistoryofaspecificitemtogetherwiththetimestampforeachmeasurement.Youcanconfigurehowmanyentriesyouwanttoseeinthecell.

Screen Thisshowsanexistingscreen.Yes,youcanembedascreenintoanotherscreenifyouwant.

Serverinfo ThisshowsasummaryofthemonitoringstatusfortheZabbixserver,suchasDBconnectivity,numberofhosts,itemsandtriggers,newvaluespersecond,andsoon.

Simplegraph Thisshowsthegraphforasingleitem,suchastheonesyoucanseeinLatestdatawithoutcreatingacustomgraph.

Simplegraphprototype

Thisislikeasimplegraph,butisforitemscreatedautomaticallyfromalow-leveldiscoveryruleprototype.

Systemstatus Thisshowsasummaryofthecurrentissues,dividedintohostgroupsandseverity.

Triggerinformation

Thisshowsasummaryoftriggerscurrentlyinaproblemstate,dividedbyseverity.Youhavetospecifyahostgroup.

Triggeroverview Thisshowseverytriggerstatusforeveryhostinaspecifichostgroup(andoptionally,application).

URL Thisshowsthecontentofanarbitrarywebpage,givenitsURL.

Everycellisalsoindependentfromtheothers:youcanbringtogetherdatabelongingtothesamehostaswellasbelongingtodifferenthostsandhosts’groups,dependingonhowyouwanttoorganizeyourscreen.

Finally,foreverycell,youcanspecifyhowmanyrowsandcolumnsitshouldspan,andforgraphiccelltypes(maps,graphs,andsoon),youcanalsodefinehowmuchspacetheyshouldtakebyspecifyingthewidthandheightinpixels.

Allthisflexibilityiscertainlypowerfulbutcanbeabitoverwhelming,soherearesomegeneralguidelinesthatyoucanrefertowhenyoucreateyourownscreens.

Averyusefultypeofscreenbringstogetherdatafromasinglehostsothatyoucanseeataglanceitsoverallperformance.You’lltypicallywanttoseesomegraphsinascreenlikethis,suchasnetworkandCPUperformance,diskusage,andanyapplication-specificgraphoritemsummaryyoumightneed,suchasdatabaseperformancegraphs,applicationserverstatistics,andsoon.

Inthefollowingexample,we’vekeptthingssimpleduetospaceconstraints,butyoucanseehowevenfourgraphscanproveusefulwhenputtogetherthisway:

www.it-ebooks.info

Aninterestingfeatureofscreencellsisthatyoucanmakethecontentdynamicbyflaggingtheaptlynamedcheckbox.Dynamiccellswillreferthesametypeofcontenttodifferenthostsdependingonthecontext.

Thismeansthatyoucancreateascreenatthetemplatelevel,flagallcellsasdynamic,andjustlikethat,everyhostinheritingthetemplatewillalsoinheritapersonalizedscreen,withallgraphsandtablesreferencingtheaforesaidhost.Thisway,youwon’thavetomanuallycreateaspecificscreenforeveryhost.

Inanothertypeofscreen,youmightwanttofocusongrouptriggersandissues.Inthiskindofscreen,atypicalcell’scontentswillbesomemaps,withhostsandlinksthatchangecolorbasedontriggerstatus,sometriggerinformationandtriggeroverviewcells,andpossiblyalogofthelatesteventsandactions.

Finally,youmightwanttocreatespecificscreensthatbringtogetherhistoricaldatafromdifferentitems,suchasapplication-specificlogfiles,outputfromexternalcommands,suchasNmap,Windowsupdatestatusforahost,andsoon.Asusual,thesky’sthelimithere.

TipKeepinmindthattheprecedingscreentypesaremerelyexamplesthatbarelyscratchthesurfaceofwhat’spossiblewithZabbix’sscreen.Youarebynomeanslimitedtothesetypes;onthecontrary,youareencouragedtomixandmatchthedifferentcellstosuityourownneeds.Don’tletusstopyoufromcreatingawesomescreens!

Onceyouhavecreatedafewscreens,thenextlogicalstepistofindawaytobringthemtogetherinanorganizedway.Slideshowsservethispurposeinaninterestingandusefulway.YoucancreateaslideshowbygoingtoConfiguration|Slideshowsandclicking

www.it-ebooks.info

onCreateslideshow.Thecreationformisprettyself-explanatory:

Muchlikeaddingitemstoacustomgraph,byclickingontheAddlinkatthebottomoftheSlideslist,youcanaddexistingscreenstotheslideshow,andyoucanreorderthembydragginganddroppingthebluearrowsnearthescreennameinthelist.Theresultwillbe,quitepredictably,aslideshowofallthescreensyouhaveputinthelist.Itwillrunoverandovercyclingthroughalltheelements.Eachslidewillhavethefocusforthenumberofsecondsequaltothedefaultdelayifyoudon’tspecifyanythingintheslide’sDelayfield.

Slideshowsareveryusefulwhenshownonabigscreeninadatacenter,butyouneedtobecarefulwhencreatingscreensthatyouknowwillendupinaslideshow.Slidesdon’tscrollvertically,soifascreenisbiggerthanthebrowserwindowusedtoshowtheslides,you’llneverbeabletoseesomeofthedata.Apossibleworkaroundistocreatescreensthatwilltakeupthewholewindowsize,butnothingmore.Thisway,you’llbesurethatallrelevantdatawillalwaysshowupontheslideshowthatyouplayonthatbigscreenyouputonthewallformonitoringpurposes.

Anotherworkaroundistomakesurethatforeachscreenbiggerthanthewindowsize,youputallimportantdataatthetopofthescreen.Thisway,someofthescreen’sdatawillshowupontheslides,whileyou’llstillbeabletoaccessallofitwhenaccessingthescreenonitsownandnotaspartoftheslideshow.

www.it-ebooks.info

SummaryInthischapter,youexploredZabbix’svisualizationfeaturesandlearnedhowtousethemtogetthemostoutofyourmonitoringdata.Sometimes,thevalueofameasurementdoesn’tlieintheeventsandactionsthatitcantrigger,butinitscorrelationwithothermeasurements,bothintime(graphs)andinstantly(maps).Thisisespeciallytruewithnetworkmonitoring,wheretheabilitytopredictthefutureneedsofanetwork,andadapttothem,isjustasimportantasactingoncontingentissues.

WehavereachedtheendofourbriefjourneythroughZabbix’sconfigurationanduse.Now,youshouldbeabletocorrectlysizeaZabbixinstallationbasedonyouenvironment;findthebestandmostappropriatetoolsandprotocolstomonitoryourdata;automatedevicediscoveryandmonitoringasmuchaspossible(andwhennottoautomateit);andmovebeyondactionsandtriggersandvisualizeallyourdatainmeaningfulways.

Withalltheseskillsunderyourbelt,weareconfidentthatyou’llbeabletoadaptapowerfulandflexibletoollikeZabbixtoyourownnetworkandnotbeconfinedtodefaulttemplatesthatmay,ormaynot,reflectyouractualmonitoringneeds.

Monitoringacomputernetworkisoftenalsoadiscoveryjourney,whereyoucangainunexpectedwisdomfromapparentlydryanduninspiringdata,suchasSNMPvaluesandserverlogs.Withthisshortbook,wehopewehaveshownyouhowZabbixcanbeanexcellentmeanstogainsuchwisdomifyouarewillingtoplaywithitforawhileandputtogooduseallitspowerfulfeatures.

www.it-ebooks.info

AppendixA.PartitioningtheZabbixDatabase

www.it-ebooks.info

MySQLpartitioningHereareallthestoredproceduresyouneedtocreatetoproperlyhandledatabasepartitioningwithMySQL.

YouneedtocreatealloftheminyourZabbixdatabase.

Notethatalltheproceduresdescribedherearealsoavailableathttps://github.com/smartmarmot/zabbix_network_monitoring/tree/master/Chapter1.

www.it-ebooks.info

Thepartition_maintenanceprocedureThisisthemostimportantprocedure,whichwillmanagealltheotherstoredproceduresinvolvedinthecreation/dropandverificationofpartitions,asfollows:

DELIMITER$$

CREATEPROCEDURE`partition_maintenance`(SCHEMA_NAMEVARCHAR(32),

TABLE_NAMEVARCHAR(32),KEEP_DATA_DAYSINT,HOURLY_INTERVALINT,

CREATE_NEXT_INTERVALSINT)

DECLAREOLDER_THAN_PARTITION_DATEVARCHAR(16);

DECLARELESS_THAN_TIMESTAMPINT;

DECLARECUR_TIMEINT;

CALLpartition_verify(SCHEMA_NAME,TABLE_NAME,HOURLY_INTERVAL);

SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(NOW(),'%Y-%m-%d

00:00:00'));

IFDATE(NOW())='2014-04-01'THEN

SETCUR_TIME=UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD(NOW(),

INTERVAL1DAY),'%Y-%m-%d00:00:00'));

ENDIF;

SET@__interval=1;

create_loop:LOOP

IF@__interval>CREATE_NEXT_INTERVALSTHEN

LEAVEcreate_loop;

ENDIF;

SETLESS_THAN_TIMESTAMP=CUR_TIME+(HOURLY_INTERVAL*

@__interval*3600);

SETPARTITION_NAME=FROM_UNIXTIME(CUR_TIME+

HOURLY_INTERVAL*(@__interval-1)*3600,'p%Y%m%d%H00');

CALLpartition_create(SCHEMA_NAME,TABLE_NAME,

PARTITION_NAME,LESS_THAN_TIMESTAMP);

SET@__interval=@__interval+1;

ENDLOOP;

SETOLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(),INTERVAL

KEEP_DATA_DAYSDAY),'%Y%m%d0000');

CALLpartition_drop(SCHEMA_NAME,TABLE_NAME,

OLDER_THAN_PARTITION_DATE);

DELIMITER;

Thisstoredprocedurewillbethecoreofourhousekeeping.Itwillbecalledwiththefollowingsyntax:

CALLpartition_maintenance('<zabbix_db_name>','<table_name>',

<days_to_keep_data>,<hourly_interval>,<num_future_intervals_to_create>)

www.it-ebooks.info

Thepartition_createprocedureThisprocedureisresponsibleforcreatingnewpartitionsacrossyourschema.Whatfollowshereistheprocedureitself:

DELIMITER$$

CREATEPROCEDURE`partition_create`(SCHEMANAMEVARCHAR(64),TABLENAME

VARCHAR(64),PARTITIONNAMEVARCHAR(64),CLOCKINT)

SCHEMANAME=TheDBschemainwhichtomakechanges

TABLENAME=Thetablewithpartitionstopotentiallydelete

PARTITIONNAME=Thenameofthepartitiontocreate

Verifythatthepartitiondoesnotalreadyexist

DECLARERETROWSINT;

SELECTCOUNT(1)INTORETROWS

FROMinformation_schema.partitions

WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAMEAND

partition_name=PARTITIONNAME;

IFRETROWS=0THEN

1.Printamessageindicatingthatapartitionwas

created.

2.CreatetheSQLtocreatethepartition.

3.ExecutetheSQLfrom#2.

SELECTCONCAT("partition_create(",SCHEMANAME,",",

TABLENAME,",",PARTITIONNAME,",",CLOCK,")")ASmsg;

SET@SQL=CONCAT('ALTERTABLE',SCHEMANAME,'.',

TABLENAME,'ADDPARTITION(PARTITION',PARTITIONNAME,'VALUESLESSTHAN

(',CLOCK,'));');

PREPARESTMTFROM@SQL;

EXECUTESTMT;

DEALLOCATEPREPARESTMT;

ENDIF;

DELIMITER;

www.it-ebooks.info

Thepartition_verifyprocedureThispartitionisresponsibleforverifyingwhetherapartitionisalreadypresent,andifitisn’t,partition_verifywillcreatethem,asfollows:

DELIMITER$$

CREATEPROCEDURE`partition_verify`(SCHEMANAMEVARCHAR(64),TABLENAME

VARCHAR(64),HOURLYINTERVALINT(11))

DECLARERETROWSINT(11);

DECLAREFUTURE_TIMESTAMPTIMESTAMP;

*Checkifanypartitionsexistforthegiven

SCHEMANAME.TABLENAME.

SELECTCOUNT(1)INTORETROWS

WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAMEAND

partition_nameISNULL;

*Ifpartitionsdonotexist,goaheadandpartitionthetable

IFRETROWS=1THEN

*Takethecurrentdateat00:00:00andaddHOURLYINTERVAL

toit.Thisisthetimestampbelowwhichwewillstorevalues.

*Webeginpartitioningbasedonthebeginningofaday.

Thisisbecausewedon'twanttogeneratearandompartition

*thatwon'tnecessarilyfallinlinewiththedesired

partitionnaming(ie:ifthehourintervalis24hours,wecould

*endupcreatingapartitionnownamed"p201403270600"

whenallotherpartitionswillbelike"p201403280000").

SETFUTURE_TIMESTAMP=TIMESTAMPADD(HOUR,HOURLYINTERVAL,

CONCAT(CURDATE(),"",'00:00:00'));

SETPARTITION_NAME=DATE_FORMAT(CURDATE(),'p%Y%m%d%H00');

—Createthepartitioningquery

SET@__PARTITION_SQL=CONCAT("ALTERTABLE",SCHEMANAME,

".",TABLENAME,"PARTITIONBYRANGE(`clock`)");

SET@__PARTITION_SQL=CONCAT(@__PARTITION_SQL,"(PARTITION

",PARTITION_NAME,"VALUESLESSTHAN(",UNIX_TIMESTAMP(FUTURE_TIMESTAMP),

"));");

—Runthepartitioningquery

PREPARESTMTFROM@__PARTITION_SQL;

EXECUTESTMT;

ENDIF;

DELIMITER;

www.it-ebooks.info

Thepartition_dropprocedureThisstoredprocedureisresponsiblefordroppingthepartitionsolderthanagivenperiod,asfollows:

DELIMITER$$

CREATEPROCEDURE`partition_drop`(SCHEMANAMEVARCHAR(64),TABLENAME

VARCHAR(64),DELETE_BELOW_PARTITION_DATEBIGINT)

SCHEMANAME=TheDBschemainwhichtomakechanges

TABLENAME=Thetablewithpartitionstopotentiallydelete

DELETE_BELOW_PARTITION_DATE=Deleteanypartitionswithnames

thataredatesolderthanthisone(yyyy-mm-dd)

DECLAREdoneINTDEFAULTFALSE;

DECLAREdrop_part_nameVARCHAR(16);

Getalistofallthepartitionsthatareolderthanthedate

inDELETE_BELOW_PARTITION_DATE.Allpartitionsareprefixed

a"p",souseSUBSTRINGTOgetridofthatcharacter.

DECLAREmyCursorCURSORFOR

SELECTpartition_name

WHEREtable_schema=SCHEMANAMEANDTABLE_NAME=TABLENAME

ANDCAST(SUBSTRING(partition_nameFROM2)ASUNSIGNED)<

DELETE_BELOW_PARTITION_DATE;

DECLARECONTINUEHANDLERFORNOTFOUNDSETdone=TRUE;

Createthebasicsforwhenweneedtodropthepartition.Also,

create

@drop_partitionstoholdacomma-delimitedlistofall

partitionsthat

shouldbedeleted.

SET@alter_header=CONCAT("ALTERTABLE",SCHEMANAME,".",

TABLENAME,"DROPPARTITION");

SET@drop_partitions="";

Startloopingthroughallthepartitionsthataretooold.

OPENmyCursor;

read_loop:LOOP

FETCHmyCursorINTOdrop_part_name;

IFdoneTHEN

LEAVEread_loop;

ENDIF;

SET@drop_partitions=IF(@drop_partitions="",

drop_part_name,CONCAT(@drop_partitions,",",drop_part_name));

ENDLOOP;

www.it-ebooks.info

IF@drop_partitions!=""THEN

1.BuildtheSQLtodropallthenecessarypartitions.

2.RuntheSQLtodropthepartitions.

3.Printoutthetablepartitionsthatweredeleted.

SET@full_sql=CONCAT(@alter_header,@drop_partitions,

PREPARESTMTFROM@full_sql;

EXECUTESTMT;

SELECTCONCAT(SCHEMANAME,".",TABLENAME)AS`table`,

@drop_partitionsAS`partitions_deleted`;

Nopartitionsarebeingdeleted,soprintout"N/A"(Not

applicable)toindicate

thatnochangesweremade.

SELECTCONCAT(SCHEMANAME,".",TABLENAME)AS`table`,"N/A"

AS`partitions_deleted`;

ENDIF;

DELIMITER;

www.it-ebooks.info

Thepartition_maintenance_allprocedureThisprocedurecallsthepartition_maintenanceprocedureforeachhistory/trendtable.Pleasenotethatforallthehistorytables,weareapplyingthesameintervals,whichare730daysoftrenddataand28daysofhistorydata.Here’showthisprocedureworks:

DELIMITER$$

CREATEPROCEDURE`partition_maintenance_all`(SCHEMA_NAMEVARCHAR(32))

CALLpartition_maintenance(SCHEMA_NAME,'history',28,24,

CALLpartition_maintenance(SCHEMA_NAME,'history_log',28,

24,14);

CALLpartition_maintenance(SCHEMA_NAME,'history_str',28,

24,14);

CALLpartition_maintenance(SCHEMA_NAME,'history_text',28,

24,14);

CALLpartition_maintenance(SCHEMA_NAME,'history_uint',28,

24,14);

CALLpartition_maintenance(SCHEMA_NAME,'trends',730,24,

CALLpartition_maintenance(SCHEMA_NAME,'trends_uint',730,

24,14);

DELIMITER;

www.it-ebooks.info

HousekeepingconfigurationAsperourexample,thehousekeepingneedstobeconfigured,asshowninthefollowingscreenshot,withahistorydatastorageperiodof730daysandatrenddatastorageperiodof28days.Here,youcanchangethosevaluesbearinginmindthatyoualsoneedtochangetheparameterpassedtothestoredprocedures.

Tochangethehousekeepingsettinginthewebinterface,yousimplyneedtogotoAdministration|General|Housekeeping(fromthedrop-downlist),andhereistheconfiguration:

www.it-ebooks.info

AppendixB.CollectingSquidMetrics

www.it-ebooks.info

SquidmetricscriptHere,youcanfindthescriptwediscussedinChapter3,MonitoringYourNetworkServices,andcreatethescriptintheusuallocation,thatis,at/home/zabbix/bin/squidcheck.sh.

Createthescriptwiththefollowingcontent:

catsquidcheck.sh

#!/bin/bash

VERSION="1.0"

functionusage()

echo"squidcheckversion:$VERSION"

echo"usage:"

echo"$0http_requests-NumberofHTTPrequestsreceived"

echo"$0clients-Numberofclientsaccessing

cache"

echo"$0icp_received-NumberofICPmessagesreceived"

echo"$0icp_sent-NumberofICPmessagessent"

echo"$0icp_queued-NumberofqueuedICPreplies"

echo"$0htcp_received-NumberofHTCPmessagesreceived"

echo"$0htcp_sent-NumberofHTCPmessagessent"

echo"$0req_fail_ratio-Requestfailureratio"

echo"$0avg_http_req_per_min-AverageHTTPrequestsperminute

sincestart"

echo"$0avg_icp_msg_per_min-AverageICPmessagesperminute

sincestart"

echo"$0request_hit_ratio-RequestHitRatios"

echo"$0byte_hit_ratio_5-ByteHitRatio5mins"

echo"$0byte_hit_ratio_60-ByteHitRatio60mins"

echo"$0request_mem_hit_ratio_5-RequestMemoryHitRatios5mins"

echo"$0request_mem_hit_ratio_60-RequestMemoryHitRatios60

echo"$0request_disk_hit_ratio_5-RequestDiskHitRatios5mins"

echo"$0request_disk_hit_ratio_60-RequestDiskHitRatios60mins"

echo"$0servicetime_httpreq-HTTPRequests(All)"

echo"$0process_mem-ProcessDataSegmentSizevia

echo"$0cpu_usage-CPUUsage"

echo"$0cache_size_disk-StorageSwapsize"

echo"$0cache_size_mem-StorageMemsize"

echo"$0mean_obj_size-MeanObjectSize"

echo"$0filedescr_max-Maximumnumberoffile

descriptors"

echo"$0filedescr_avail-Availablenumberoffile

descriptors"

########

#Main#

########

www.it-ebooks.info

if[[$#!=1]];then

#NoParameter

case$1in

"http_requests")

value="`squidclientmgr:info|grep'NumberofHTTPrequests

received:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"clients")

value="`squidclientmgr:info|grep'Numberofclientsaccessing

cache:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"icp_received")

value="`squidclientmgr:info|grep'NumberofICPmessages

rval=$?;;

"icp_sent")

value="`squidclientmgr:info|grep'NumberofICPmessages

sent:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"icp_queued")

value="`squidclientmgr:info|grep'NumberofqueuedICP

replies:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"htcp_received")

value="`squidclientmgr:info|grep'NumberofHTCPmessages

rval=$?;;

"htcp_sent")

value="`squidclientmgr:info|grep'NumberofHTCPmessages

sent:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"req_fail_ratio")

value="`squidclientmgr:info|grep'Requestfailureratio:'|cut-

d':'-f2|tr-d'\t'`"

rval=$?;;

"avg_http_req_per_min")

value="`squidclientmgr:info|grep'AverageHTTPrequestsperminute

sincestart:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"avg_icp_msg_per_min")

value="`squidclientmgr:info|grep'AverageICPmessagesperminute

sincestart:'|cut-d':'-f2|tr-d'\t'`"

rval=$?;;

"request_hit_ratio")

value="`squidclientmgr:info|grep'RequestHitRatios:'|cut-d':'-

f3|cut-d','-f1|tr-d'%'`"

rval=$?;;

"byte_hit_ratio_5")

value="`squidclientmgr:info|grep'Hitsas%ofbytessent:'|awk

-F'[:,%]''{print$10}'|tr-d'\t'`"

rval=$?;;

"byte_hit_ratio_60")

value="`squidclientmgr:info|grep'Hitsas%ofbytessent:'|awk

www.it-ebooks.info

-F'[:,%]''{print$15}'|tr-d'\t'`"

rval=$?;;

"request_mem_hit_ratio_5")

value="`squidclientmgr:info|grep'Hitsas%ofallrequests:'|

awk-F'[:,%]''{print$10}'|tr-d'\t'`"

rval=$?;;

"request_mem_hit_ratio_60")

value="`squidclientmgr:info|grep'Hitsas%ofallrequests:'|

awk-F'[:,%]''{print$15}'|tr-d'\t'`"

rval=$?;;

"request_disk_hit_ratio_5")

value="`squidclientmgr:info|grep'Diskhitsas%ofhit

requests:'|awk-F'[:,%]''{print$11}'|tr-d'\t'`"

rval=$?;;

"request_disk_hit_ratio_60")

value="`squidclientmgr:info|grep'Diskhitsas%ofhit

requests:'|awk-F'[:,%]''{print$16}'|tr-d'\t'`"

rval=$?;;

"servicetime_httpreq")

value="`squidclientmgr:info|grep'HTTPRequests(All):'|cut-d':'

-f2|tr-s''|awk'{print$1}'`"

rval=$?;;

"process_mem")

value="`squidclientmgr:info|grep'ProcessDataSegmentSizevia

sbrk'|cut-d':'-f2|awk'{print$1}'`"

rval=$?;;

"cpu_usage")

value="`squidclientmgr:info|grep'CPUUsage:'|cut-d':'-f2|tr-d

'%'|tr-d'\t'`"

rval=$?;;

"cache_size_disk")

value="`squidclientmgr:info|grep'StorageSwapsize:'|cut-d':'-

f2|awk'{print$1}'`"

rval=$?;;

"cache_size_mem")

value="`squidclientmgr:info|grep'StorageMemsize:'|cut-d':'-

f2|awk'{print$1}'`"

rval=$?;;

"mean_obj_size")

value="`squidclientmgr:info|grep'MeanObjectSize:'|cut-d':'-

f2|awk'{print$1}'`"

rval=$?;;

"filedescr_max")

value="`squidclientmgr:info|grep'Maximumnumberoffile

descriptors:'|cut-d':'-f2|awk'{print$1}'`"

rval=$?;;

"filedescr_avail")

value="`squidclientmgr:info|grep'Availablenumberoffile

descriptors:'|cut-d':'-f2|awk'{print$1}'`"

rval=$?;;

exit1;;

if["$rval"-eq0-a-z"$value"];then

www.it-ebooks.info

rval=1

if["$rval"-ne0];then

echo"ZBX_NOTSUPPORTED"

echo$value

www.it-ebooks.info

IndexA

actionconditionssection/FindinghoststheZabbixwayactiondefinitionsection/Definingactionconditionsactionoperationssection/FindinghoststheZabbixwayApache

modules/ApachemonitoringApachemonitoring

about/Apachemonitoringperforming/Apachemonitoring

architectures,Zabbixabout/Zabbixarchitectures

www.it-ebooks.info

Ccomplexmaps

issues/Maps–aquicksetupforalargetopologyCPULoadparameter/Apachemonitoringcustomgraphs

creating/Creatingcustomgraphs

www.it-ebooks.info

Ddatabase

installing/Installingadatabasesize,considering/Consideringthedatabasesizeitems/Consideringthedatabasesizerefreshrate/Consideringthedatabasesizespace/ConsideringthedatabasesizeMySQLpartitioning/MySQLpartitioning

dataflow,Zabbixabout/UnderstandingZabbixdataflow

datatypes,SNMPabout/GettingdatatypesrightURL/GettingdatatypesrightINTEGER/GettingdatatypesrightSTRING/GettingdatatypesrightOID/GettingdatatypesrightIpAddress/GettingdatatypesrightCounter32/GettingdatatypesrightGauge32/GettingdatatypesrightCounter64/GettingdatatypesrightTimeTicks/Gettingdatatypesright

digabout/DNS–responsetime

discoveryitemsabout/Low-leveldiscovery

discoveryrulesabout/Low-leveldiscovery

DNSmonitoringabout/MonitoringtheDNSperforming/MonitoringtheDNSresponsetime,monitoring/DNS–responsetimeDNSSECzonerollover,monitoring/DNSSEC–monitoringthezonerollover

DNSSECparametersabout/DNSSEC–monitoringthezonerollover

www.it-ebooks.info

Ggraph

putting,onscreen/Puttingeverythingtogetherwithscreens

www.it-ebooks.info

Hhostgroups

about/Hostsandhostgroupsroutersgroup/Hostsandhostgroupsswitchesgroup/Hostsandhostgroupssubnetgroup/Hostsandhostgroups

hostsabout/UnderstandingZabbixhostsinterfaces/Hostinterfacesinventory/Hostinventory

housekeepingconfigurationabout/Housekeepingconfiguration

www.it-ebooks.info

IICMPechochecks

about/Simplechecksinterfaces/HostinterfacesInternetProtocolFlowInformationeXport(IPFIX)/Gettingnetflowfromthedevicestothemonitoringserver

www.it-ebooks.info

Llow-leveldiscovery

about/Low-leveldiscoveryadvantage/Low-leveldiscoveryrules,creating/Low-leveldiscoveryrules,managing/Low-leveldiscovery

www.it-ebooks.info

complexmaps/Maps–aquicksetupforalargetopologyDOTcreation,automating/Maps–automatingtheDOTcreationdrafting,fromDOT/DraftingZabbixmapsfromDOTputting,onscreen/Puttingeverythingtogetherwithscreens

MIBsabout/FindingtherightOIDstomonitor

MySQLpartitioningabout/MySQLpartitioningbenefits/MySQLpartitioningstoredprocedures/MySQLpartitioningpartition_maintenanceprocedure/Thepartition_maintenanceprocedurepartition_createprocedure/Thepartition_createprocedurepartition_verifyprocedure/Thepartition_verifyprocedurepartition_dropprocedure/Thepartition_dropprocedurepartition_maintenance_allprocedure/Thepartition_maintenance_allprocedure

www.it-ebooks.info

Nnetflow

about/Gettingnetflowfromthedevicestothemonitoringserverdata,gettingintoZabbix/Gettingnetflowfromthedevicestothemonitoringserverdata,receivingonserver/Receivingnetflowdataonyourserver

networkdiscoveryhosts,finding/FindinghoststheZabbixwayactionconditions,defining/Definingactionconditionsactionoperations,selecting/Choosingactionoperationsremotecommands,executing/Remotecommands

networkinterfacesabout/Low-leveldiscovery

networkservicesDNS,monitoring/MonitoringtheDNSApache,monitoring/ApachemonitoringNTP,monitoring/NTPmonitoringSquid,monitoring/Squidmonitoring

NetworkXURL/Maps–automatingtheDOTcreationabout/Maps–automatingtheDOTcreation

Nfdumpabout/Receivingnetflowdataonyourservernfcapd/Receivingnetflowdataonyourservernfdump/ReceivingnetflowdataonyourserverURL,fornfdumppackage/Receivingnetflowdataonyourserver

Nmap/ChoosingactionoperationsNTPmonitoring

about/NTPmonitoringperforming/NTPmonitoring,NTP–whatarewemonitoring?Delay/NTP–whatarewemonitoring?Offset/NTP–whatarewemonitoring?Jitter/NTP–whatarewemonitoring?

www.it-ebooks.info

finding,formonitoring/FindingtherightOIDstomonitorabout/FindingtherightOIDstomonitormapping,toZabbixitems/MappingSNMPOIDstoZabbixitems

www.it-ebooks.info

Ppartition_createprocedure

about/Thepartition_createprocedurepartition_dropprocedure

about/Thepartition_dropprocedurepartition_maintenanceprocedure

about/Thepartition_maintenanceprocedurepartition_maintenance_allprocedure

about/Thepartition_maintenance_allprocedurepartition_verifyprocedure

about/Thepartition_verifyprocedurePerlmodules

about/DNSSEC–monitoringthezonerolloverproxiesdataflow,Zabbix

about/UnderstandingtheZabbixproxies’dataflowProxyConfigFrequency=parameter

about/UnderstandingtheZabbixproxies’dataflowProxyDataFrequency=parameter

about/UnderstandingtheZabbixproxies’dataflowpyzabbix

about/RemotecommandsURL/Remotecommands

www.it-ebooks.info

Qquery_apachestats.py/Apachemonitoring

www.it-ebooks.info

RReadingRequestparameter/ApachemonitoringReqPerSecparameter/Apachemonitoringrollstateplugin

about/DNSSEC–monitoringthezonerollover

www.it-ebooks.info

Sscreen

about/Puttingeverythingtogetherwithscreenscreating/Puttingeverythingtogetherwithscreensmaps,puttingon/Puttingeverythingtogetherwithscreensgraph,puttingon/Puttingeverythingtogetherwithscreens

SiegeURL/Apachemonitoring

simplechecksabout/SimplechecksIcmpping/SimplechecksIcmppingloss/SimplechecksIcmppingsec/SimplechecksNet.tcp.service/SimplechecksNet.tcp.service.perf/Simplechecksconfiguring/Simplechecks

slideshowcreating/Puttingeverythingtogetherwithscreens

SNMPabout/KeepingSNMPsimpledata,gettingintoZabbix/GettingSNMPdataintoZabbixOIDs,findingformonitoring/FindingtherightOIDstomonitorOIDs,mappingtoZabbixitems/MappingSNMPOIDstoZabbixitemsdatatypes/Gettingdatatypesrightnetflowdata,receivingonserver/Receivingnetflowdataonyourserverlogfile,monitoringwithZabbix/MonitoringalogfilewithZabbix

SNMPgetsabout/KeepingSNMPsimple

snmptrapdabout/Snmptrapd

SNMPtrapsabout/KeepingSNMPsimple,SNMPtrapssnmptrapd/Snmptrapdtransforming,intoZabbixitem/TransformingatrapintoaZabbixitemnetflow,gettingfromdevices/Gettingnetflowfromthedevicestothemonitoringserver

Squidabout/SquidmonitoringURL/Squidmonitoring

Squidmetricscriptabout/Squidmetricscript

Squidmonitoringperforming/Squidmonitoring

www.it-ebooks.info

StartProxyPollers=parameterabout/UnderstandingtheZabbixproxies’dataflow

www.it-ebooks.info

TTCP/IPconnectionchecks

about/Simplecheckstriggerinformationcell/Puttingeverythingtogetherwithscreenstriggeroverviewcell/Puttingeverythingtogetherwithscreens

www.it-ebooks.info

Vvaluemaps

about/Gettingdatatypesright

www.it-ebooks.info

WWaitingForConnectionparameter/ApachemonitoringWebGUIinterface

installing/InstallingtheWebGUIinterface

www.it-ebooks.info

Xxdot.py

URL/Maps–automatingtheDOTcreationxml2

about/MonitoringtheDNS

www.it-ebooks.info

ZZabbix

architectures/Zabbixarchitecturesdataflow/UnderstandingZabbixdataflowproxiesdataflow/UnderstandingtheZabbixproxies’dataflowinstalling/InstallingZabbixdatabase,installing/Installingadatabasehosts/UnderstandingZabbixhostshostgroups/Hostsandhostgroups

Zabbixagentpackage,forLinuxOSURL/CreatingaZabbixagentpackagewithCheckInstall

Zabbixagentsabout/GoingbeyondZabbixagentssimplechecks/SimplechecksSNMP/KeepingSNMPsimpleSNMPtraps/SNMPtraps

ZabbixApacheUpdaterplugin/ApachemonitoringZabbixinstallation

about/InstallingZabbixinstalling,frompackages/InstallingfrompackagesZabbixagent,settingup/SettingupaZabbixagentZabbixagentpackage,creatingwithCheckInstall/CreatingaZabbixagentpackagewithCheckInstallserverconfiguration/Serverconfiguration

Zabbixproxyinstalling/InstallingaZabbixproxy

zapacheplugin/ApachemonitoringURL/Apachemonitoring

zonestatepluginabout/DNSSEC–monitoringthezonerollover

www.it-ebooks.info

zabbix network monitoring essentials - omid-online.com · table of contents zabbix network...

Documents

zabbix 4.0 and beyond -...

zabbix cas d'usage - sib - rennes monitoring - zug

security-related monitoring with zabbix › files ›...

zabbix - enterprise-class monitoring …...monitoring...

storage devices zabbix monitoring

zabbix 101 - enterprise monitoring doesn't have to suck

zabbix in japan - liaa · 3 what is zabbix? an image of...

zabbix network monitoring essentials - omid- · pdf...

zabbix monitoring

host & container monitoring - oodrive€¦ · container...

zabbix: monitoring millions of metrics and a zabbix 3.0 ......

monitoring docker -...

enterprise-class monitoring solution for everyone · about...

enterprise-class monitoring solution for … · les types...

【zabbix 2.1】trial of vm monitoring function of zabbix...

nodeconductor zabbix documentation - read the docs · pdf...

zabbix network monitoring essentials - sample chapter

zabbix monitoring -...

alexei vladishev - open source monitoring with zabbix

open source enterprise monitoring with zabbix - · pdf...