dhruva kumar bhagi sr. software engineer · 2017-10-08 · sr. software engineer splunk inc....
TRANSCRIPT
![Page 1: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/1.jpg)
Copyright©2016SplunkInc.
DhruvaKumarBhagiSr.SoftwareengineerSplunkInc.
Replicationofsummarydatainindexercluster
![Page 2: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/2.jpg)
Disclaimer
2
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
![Page 3: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/3.jpg)
Indexerclustertopology
3
MasterSearchhead
FORWARDERS
Indexer Indexer IndexerForReplication
Forgenerationinfo
ForSearch
Forwardingdatatoindexers
Master-slavecommunication
Forreplication
Searchhead-idx
![Page 4: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/4.jpg)
Summaries
4
Summaries arehigherlevelrepresentationofthesearchresultsResideonindexertohelpacceleratetheexecutionoflongrunningsearchesoriginatingfromsearchheadsApplicableforstatisticalsearcheslike‘index=main|statscount’
![Page 5: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/5.jpg)
Summaries
5
Twotypesofsummaries:• Reportaccelerationsummaries– Generatedbyacceleratedreports/searches• Datamodelsummaries– Generatedbyaccelerateddatamodelpivotsearches
ResideunderINDEX/{datamodel_summary|summary}/BID/SH_ID/summary_name/…Note:summariesaredifferentfromsummaryindex
![Page 6: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/6.jpg)
Problem
6
SummarieswerepreviouslyneverreplicatedIfanindexerwithasummarygoesdown,bucketprimariesmovetoanothersearchablecopy,andsearcheswillnothaveaccesstothesummaries(untiltheygetregenerated),therebysearchesrunslow.Regeneratingsummariescantakelotoftime
![Page 7: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/7.jpg)
Replicatingsummaries
7
Fromversion6.4,SplunkcanreplicateDA&RMsummariesinindexerclusterenvironmentPeersmaintainthelistoftheirsummaries(justlikebuckets)andreportsthemtoCMduringregistration(or)onanystatechangeIfanypeerismissingasummaryforabucket,masterschedulesasummaryreplicationwiththispeerasareplicationtargetOnlyreplicatesummariesforwarm/coldbuckets(nothot)services/cluster/master/buckets/BUKCETlistssummariesofeachbucket
![Page 8: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/8.jpg)
Summariesforhotbuckets
8
Forhotbuckets,summarysearchesfromsearchheadshitallthesearchablecopies(insteadofjustprimariesfornormalsearches)therebylettingindexersbuildthesummariesindividuallyforhotbuckets.Noneedforsummaryregenerationevenonindividualnodefailures.Runningsummarygeneratingsearchesonallsearchablecopiesofahotbucketintroducesmoreresourceusage(longersearchprocess,extramemoryusage)!
![Page 9: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/9.jpg)
VolumeRetention
9
Summariescanbesetupinavolumetomanagespaceusage
Ifindexerexceedsthevolumespace(maxVolumeDataSizeMB),splunkstartstrimmingsummaries– Atrimmedsummaryleavestombstoneindicatingthetrimmedstate– Tombstoneavoidsregeneratingsummariesinthesubsequentsearchruns– Oncewetrimasummary,wepropagatethisinfototheCM,andtheCMwill
issueatrimtoallothercopiesofthesummary.
![Page 10: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/10.jpg)
Configuration
10
Toturnonsummaryreplication,makesummary_replication=trueunderclusteringstanzaonclustermaster.Bydefaultsummaryreplicationisturnedoff.max_peer_sum_rep_load (defaultsto5)configureshowmanymaximumsummaryreplicationsperpeerConfigchangesarereloadable(i.e.doesnotrequireasplunkrestart)
![Page 11: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/11.jpg)
THANKYOU
![Page 12: Dhruva Kumar Bhagi Sr. Software engineer · 2017-10-08 · Sr. Software engineer Splunk Inc. Replication of summary data in indexer cluster. Disclaimer 2 During the course of this](https://reader033.vdocuments.net/reader033/viewer/2022052303/5f1d859a8537fa6aad6aa0ba/html5/thumbnails/12.jpg)
Copyright©2016SplunkInc.
````
Wednesday,Sep283:00PM- 3:15PM