elas%csearch meetup #5files.meetup.com/7646592/20150916-human-bot-statistics-with-elk.pdf ·...
TRANSCRIPT
• running13websites(www.jobs.ch,www.jobup.ch,…)•migra%ngservicesfromalegacyenvironmenttonewinfrastructure
Ourcurrentsetup
7
•newenvironmentbasedonlxccontainers
•42produc%oncontainers(app/search/cache/db/…)• similarsetupforstagingenvironment
• ahugeloadofdistributedlogfileseverywhere
Ourcurrentsetup
8
Ourcurrentsetup
9
app01 app02 app03
logstash logstash logstash
redis01
logstash
elasticsearch
• severallogfilesources• logstashforwarderoneachnode•oneredisclusterforcatchinglogs•onelogstashtopushlogstoelas%csearch
data!data!data!
13
• Jobcloud\TrackerBundle*writestoX-Custom-DataHTTPHeader
•HTTPHeaderiswriYentoaccesslog,alongwithsomeothers
•X-Custom-Datagetsremovedbyreverseproxy
*notopensourcedyet
data!data!data!
16
•uselogstashfiltertoenhancethelogfileinforma%on
• logstash-filter-tordetect*:gathersinforma%onabouttornetworks
• logstash-filter-hitclassifica%on*:addsinforma%onabouttheuser(human,bot,headless)
*notopensourcedyet
data!data!data!
17
• store4weeksofdatainproduc%on•moveeverythingelsetoAmazonS3
• runain-housekibanasetuptohandlelongtermrequests
• “measurestuff,evenifyoudon’tneeditrightnow”—itsmoreexpensivetonothavethisdata
*notopensourcedyet
• centralisedlogviewusingkibana•deeperunderstandingforopera%ons• insightsfordevelopment
•dashboardsforpmormarke%ng
data!nowwhat?
19
•Whichkeywordwasusedtofindaspecificad?
•HowmanyusersviewedtheNovar%sjobads,reques%ngfromBasel?
•Whichbotsarethemostpainfulonsearch?
•Howodenwasanaddisplayed,andonwhichavg.posi%on?
data!nowwhat?
20