experian customer presentation
Post on 06-Jan-2017
91 Views
Preview:
TRANSCRIPT
2
DisclaimerDuringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.For
importantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmade
asofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesorfunctionality
describedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
ItshouldalsobenotedthattheviewsexpressedinthispresentationaresolelythoseoftheauthorinhisprivatecapacityanddonotinanywayrepresenttheviewsofConsumerInfo.com,Inc.(aka:ExperianConsumerServices),
anyotherentityofExperian,oritsAffiliates.
Alllogosusedinthispresentationarepropertyoftheirrespectivecompanies.
3
AboutMe• MikeSclimenti,SeniorSystemsEngineer
• ExperianConsumerServices– ITSystemsAdministration/Engineeringfor20+years
ê HighlyScalableInfrastructureDeployments&DisasterRecoveryê LargeScaleVMware&Symantec(Veritas)NetBackup Environmentsê ApplicationDeployments,SystemsManagement,ActiveDirectory,etc.
– MonitoringSystems2+years
• Splunkcustomer– Userfor8years– Adminfor2years(Splunk6.1,6.3)
• FavoriteSplunktee-shirt:“Becauseninjasaretoobusy”
4
Agenda• Architecture&LessonsLearneddeployingSplunkCloud:
– S3viatheSplunkAppforAWS– Kinesis– LambdaFunctions– TheHTTPEventCollector
• Howwewentfrom15minutesoflatencyonproductiondashboardsto…– Sub-5secondsoflatencysendinglogsdirectlyfromKinesis(viaLambda)tothe
HTTPEventCollector
6
Splunk’s S3ConnectorTheS3Connectorisefficientfor:
⏤ CloudFront
⏤ ELB(ElasticLoadBalancer)
⏤ CloudWatch &CloudWatch Logs
⏤ Cloudtrail
⏤ Billing
7
TheS3ConnectorWasWorking,But…• ThenIwentto.conf 2015• So,asIwassittingintheKeynotesessiononDay1,Ithought:
– CouldIgodirectlytotheHTTPEventCollectorfromtheapplication?ê NomoreUniversalForwarderstoinstallorupdateê FeweragentsrunningontheEC2instances
– WouldloggingtoKinesisandthentotheHTTPEventCollectorbemoreefficient?
AmazonEC2 AmazonKinesis AmazonLambda
8
TheHTTPEventCollector
Applications IoT Devices
Agentless,directdataonboardingviaastandarddeveloperAPI
curl -k https://<host>:8080/services/collector -H ‘Authorization: Splunk <token>’ -d ‘{”event”:”Hello Event Collector”}’
9
TheHTTPEventCollector(cont.)• Gotbacktotheoffice,begandoingfurtherresearch
• StartedplanningmigrationfromS3ConnectortotheHTTPEventCollector
• Beganseeinglatencyissuesw/theingestfromS3whilerunningsomeloadtests
• Timelineformigrationacceleratedduetolatencyof15minutesingestinglogsfromS3
Butthen…IrealizedHOUSTONWEHAVEAPROBLEM!
10
TheHTTPEventCollector(cont.)
10
• WewererunningSplunkCloudversion6.2• TheHTTPEventCollectordidnotexistinSplunkCloudversion6.2• InstalledtheHTTPEventCollectoronaHeavyForwarderrunningtheSplunkEnterprise6.3.
AmazonEC2 AmazonKinesis AmazonLambda SplunkEnterprise6.3HeavyForwarder
SplunkCloud6.2
11
TheHTTPEventCollector(cont.)
• Everythingwasrunninggreatuntiluntilwecrankedupourtraffic…
• LuckilySplunkCloudmadeversion6.3availableforproduction
• SplunkCloud6.2wasupgradedto6.3
• HTTPEventCollectorwasenabledonindexers
• Lambdafunctionsupdated
• Tuningbegan...
15
LambdaBatchSize• BatchsizeisthemaxnumberofeventsthatsentforsingleinvocationoftheLambdafunction
• Increaseditfrom100to1000to5000to10000thenbackto5000• 646bytesaverageeventsizebutthenHTTPeventcollectorstartedtoerrorsometimesbecauseofthedefaultmax_content_length =1,000,000bytes
• 1,000,000/646=1548eventsinbatch
sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-24h latest=now | eval event_size=len(_raw) | stats avg(event_size
17
HTTPEventCollectorScalingLimits.conf[http_input]max_content_length =1000000(bytes)
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf
Increasethemax_content_length =5,000,000bytes(~5MB)Batchsize=5000,memoryfortheLambdaat512MB
18
LambdaTuning• Makesureyouusehttps/SSLbetweenLambdaandHTTPEventCollector
• Setanappropriatebatchsize!“1000”isbetterthan“100”
• SetLambdaFunctionto“Latest”NOT “TrimHorizon”
• GiveyourLambdafunctiontherightamountofmemory• Changethetimeoutfrom“10”to“30”
19
AWSKinesisShards• Eachshardcansupport:
– Upto5transactionspersecondforreads– Uptoamaxtotaldatareadrateof2MB/sec– Upto1Krecordspersecondforwrites– Uptoamaxtotaldatawriterateof1MB/secê 2MB/secpershardê Planforpeaks
• MakesureyousplitKinesisintoenoughshardssothatitcanhandle:– Inboundstreamsfromyourapplication– OutboundstreamstoS3and/ortheHTTPEventCollector
20
MeasuringOurProgress• LatencySearch
sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-2mlatest=now|eval latency_in_seconds=(_indextime - _time)|statsperc80(latency_in_seconds)as80th_percentile_latency_in_seconds
22
ThingstoRemember• S3worksbuttheHTTPEventCollectorisfaster
• YoumustbeusingSplunk CloudORSplunk Enterprise6.3(orhigher)
• TuneyourLambdafunction(mayimpactyourfunction$$$)
• ScaleupyourHTTPEventCollector
• MakesureyouhaveenoughKinesisshards(mayimpactyourKinesis$$$)
• Measureyourprogressthroughdashboardsandalerts
23
Resources• .conf2015“TheGreatShakeOff”
– http://www.ustream.tv/recorded/73893599 (startsatthe22minmark)
• Splunk’s HTTPEventCollector– http://dev.splunk.com/view/event-collector/SP-CAAAE6M
• AWSLambda– http://docs.aws.amazon.com/lambda/latest/dg/welcome.html
• AWSKinesisShardLimits– http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
top related