pan109098 - going splunking: using splunk for server log
TRANSCRIPT
![Page 1: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/1.jpg)
PAN109098- GoingSplunking:UsingSplunkforserverloganalyticsinBlackboardLearn7/24/17(Monday)11:00AM- 284-285
PRESENTEDBY:CHRISBRAYBLACKBOARDADMINISTRATORATUNIVERSITYOFARKANSAS|IANGOHSOFTWAREENGINEERATJOHNSHOPKINSUNIVERSITY
![Page 2: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/2.jpg)
Statementsregardingourproductdevelopmentinitiatives, includingnewproductsandfutureproductupgrades,updatesorenhancementsrepresentourcurrent intentions,butmaybemodified,delayedorabandonedwithout priornotice andthere isnoassurancethatsuchoffering,upgrades,updatesorfunctionalitywillbecomeavailableunlessanduntiltheyhavebeenmadegenerallyavailabletoourcustomers.
![Page 3: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/3.jpg)
TheProblem
![Page 4: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/4.jpg)
TheProblem
Goodmorning,
We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.
Pleaseadvise. Thanks!
blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT
Admins- Howoftendoyougetthisemail?
![Page 5: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/5.jpg)
TheProblem
• Whatstepsdoyounormallytaketoanalyzetheissue?
•
AskTheAudience
![Page 6: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/6.jpg)
TheProblem
1. Manylogs
2. LogFormats– lackconsistentformats,timeformats
3. MultipleServers– needtocentralizelogs
4. Expertise/Access– whocanaccessthelogs,whounderstandswhichlogdoeswhat?
![Page 7: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/7.jpg)
Splunk
![Page 8: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/8.jpg)
WhatisSplunk?https://www.splunk.com/
• “GoogleforLogfiles”-- HelgeKlein/https://helgeklein.com/blog/2014/09/splunk-work/
• “Schemaonthefly”
• Splunk EnterpriseCoreFeatures:
– CollectandIndexData
– Search,AnalyzeandVisualize
– Monitor,AlertandReport
• ProvidesPremiumSolutions:Security, ITServiceIntelligence,UserBehaviorAnalytics
![Page 9: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/9.jpg)
Components ofSplunk
• Forwarders - Splunk softwarecaningestallkindsofdatatypesandsources.File-baseddatacanbesentvia forwarders thatresidedirectlyonthedatasources
• Indexer - AnindexeristheSplunk instancethatindexesdata.Theindexertransformstherawdataintoeventsandstorestheeventsintoanindex.
• SearchHead- Inadistributedsearchenvironment,thesearchheadistheSplunkinstancethatdirectssearchrequeststoasetofsearchpeersandmergestheresultsbacktotheuse
![Page 10: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/10.jpg)
Managed&SelfHostedBlackboardEnvironments
ChrisBray/U.Arkansas–ManagedHosting• logsarezippedandsentoverdaily
• Splunk forwardingagenttransferslogsintolocalinstanceofSplunk
I.Goh/JohnsHopkins University- SelfHosted• useForwarderstosendlogstolocal instanceofSplunk
• prodwebservers(andourdev,testserversaswell)sendreal-timedata
![Page 11: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/11.jpg)
WhataboutSAAS?
OfferingELK(Elasticsearch,Logstash,andKibana)?• PossiblylinktosomeoneelsedoingaDevCon onELK?
![Page 12: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/12.jpg)
Alternatives toSplunk
• MS SystemCenterOperationsManager(SCOM) (Windows)
• Nagios (OpenSource)
• ELK(Elasticsearch,Logstash,andKibana)
![Page 13: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/13.jpg)
BlackboardLogs– whatdoyousend,howlongisitkept/indexed
C.Bray /U.Arkansas- Manage Hosted• Blackboard
• tomcat/access-logs
• bb-email.log
• bb-authentication.log
I.Goh /JohnsHopkins University- SelfHosted• MicrosoftIIS(sourcetype: iis)
• Blackboard
• bb-services-log.txt (sourcetype:bb_services - mightbejust acustomlabelforus)
• tomcat/bb-access-log- txt(sourcetype:access_combined_wcookie)
• tomcat/stdout-stderr- log(sourcetype: log4j)
• Productionindexkeepssixmonthsofdata
![Page 14: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/14.jpg)
Examples
![Page 15: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/15.jpg)
SearchExamples
HeretherebeExampleshttp://bbadmin.uark.edu
![Page 16: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/16.jpg)
ReportExamples
• StartupTimesofwebapp servers
• Search:host=hostname*source=“tomcat\\stdout-stderr-*.log""Blackboardapplicationserverreadytoacceptrequests”
• Report:
![Page 17: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/17.jpg)
ReportExamples
• Geolocation:visualizewheretheBbStudentappishittingusfrom(beforewegetthemedu.comrequests)
• Search:host=hostname*"/webapps/Bb-mobile-bb_bb60/customAuthSuccess"|iplocation c_ip |geostats countBYc_ip
• Report:
![Page 18: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/18.jpg)
AlertExamples
• Useareal-timealerttomonitoreventsoreventpatternsastheyhappen.Youcancreatereal-timealertswithper-resulttriggeringorrollingtimewindowtriggering
• WouldnotbeusefulintheManageHostedsituationiflogswillbedelayed(duetotransfer)
• JHU:weuseSCOMforalerts
• Example:theStartupTimesearchcouldbeusedtotriggeranemail/pager(orevenaPhilipsHuelight!)
![Page 19: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/19.jpg)
UsesofSplunk outsideofIT
• UNLV– usingSplunk toanalyzelearningdataw/machinelearning
• https://www.splunk.com/en_us/resources/video.UzaWVuNjE60_AMjGA_NfnDfE2FGoIIFB.html#
• “Thediscoveryandminingofsuch(LMS)logsledhimtobuildadatadictionarythatenabledhimtoidentifytheevents,classifythem,andgaininsightsintotheactionsstudentswerelikelytotakeandwhichonespredictedtheirachievement“
![Page 20: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/20.jpg)
Splunk Resources
![Page 21: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/21.jpg)
GettingStartedwithSplunk
• https://www.splunk.com/en_us/download.html
• WhenyoudownloadSplunk Enterpriseforthefirsttime,yougetanEnterpriseTriallicensefor60days.ThisEnterpriseTriallicenseincludesallofthefeatures,butlimitstheamountofdatathatyoucanindexeachday.Thedailylimitis500MB.
• SearchTutorial-http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial
• QuickReferenceGuide- https://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-quick-reference-guide.pdf
• Splunk Community- https://www.splunk.com/en_us/community.html
• SplunkLive!- http://splunklive.splunk.com/ (it’safreeevent!)
![Page 22: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/22.jpg)
Splunk Free
• http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/MoreaboutSplunkFree
– Splunk Freeisdesignedforpersonal,adhocsearchandvisualizationofITdata.YoucanuseSplunkFreeforongoingindexingofsmallvolumes(<500MB/day)ofdata.Additionally,youcanuseitforshort-termbulk-loadingandanalysisoflargerdatasets--Splunk Freeletsyoubulk-loadmuchlargerdatasetsupto3timeswithina30dayperiod.Thiscanbeusefulforforensicreviewoflargedatasets.
![Page 23: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/23.jpg)
Don’tforgettoratethissessionintheBbWorldapp.
![Page 24: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/24.jpg)
Oh?TheSolution…
![Page 25: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/25.jpg)
TheProblem
Goodmorning,
We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.
Pleaseadvise. Thanks!
blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT
Admins- Howoftendoyougetthisemail?
![Page 26: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/26.jpg)
TheProblem– SolvingitwithSplunk 1
1. Searchfor‘4e1f842a-6f34-4657-8c76-d3d50bbb73a5’
![Page 27: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/27.jpg)
TheProblem– SolvingitwithSplunk 2
2. Searchfor‘java.lang.NoClassDefFoundError’, overthelastsevendays
![Page 28: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/28.jpg)
TheProblem– SolvingitwithSplunk 345
3. Seeit’sonlyappearingononehost(webapp)server
4. Takeserveroutofloadbalancer,restartservices.
5. Monitorforanyotherjava.lang.NoClassDefFoundError afterrestart
![Page 29: PAN109098 - Going Splunking: Using Splunk for server log](https://reader034.vdocuments.net/reader034/viewer/2022052700/628e82aff0ae4f199d70bbdb/html5/thumbnails/29.jpg)