Download - Machine Data 101
![Page 1: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/1.jpg)
Copyright©2014SplunkInc.
MachineData101
GaryBurgettSr.SE
11/1/2016
![Page 2: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/2.jpg)
WhatDoesMachineDataLookLike?Sources
OrderProcessing
CareIVR
MiddlewareError
2
![Page 3: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/3.jpg)
MachineDataContainsCriticalInsightsCustomerID OrderID
Customer’sTweet
TimeWaitingOnHold
TwitterID
ProductID
Company’sTwitterID
CustomerIDOrderID
CustomerID
Sources
OrderProcessing
CareIVR
MiddlewareError
3
![Page 4: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/4.jpg)
MachineDataContainsCriticalInsightsOrderID
Customer’sTweet
TimeWaitingOnHold
ProductID
Company’sTwitterID
OrderID
CustomerID
TwitterID
CustomerID
CustomerID
Sources
OrderProcessing
CareIVR
MiddlewareError
4
![Page 5: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/5.jpg)
StructuredRDBMS
SQL Search
SchemaatWrite SchemaatRead
Traditional Splunk
SplunkApproachtoMachineData
Copyright © 2014 Splunk Inc. 5
ETL UniversalIndexing
Volume Velocity Variety
Unstructured
![Page 6: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/6.jpg)
Splunk:ThePlatformforMachineData
6
DeveloperPlatform
Reportand
analyze
Customdashboards
Monitorandalert
Adhocsearch
OnlineServices
WebProxy
DataLossPrevention
Storage Desktops
PackagedApplications
CustomApplications
Databases
CallDetailRecords
SmartphonesandDevices
FirewallAuthentication
Fileservers
Endpoint
ThreatIntelligence
Asset&CMDB
Employee/HRInfo
DataStoresApplications
ExternalLookups
Badgingrecords
Emailservers
VPN
Anyamount,anylocation,anysource
Schema-on-the-fly
Universalindexing
Noback-endRDBMS
Noneedtofilterdata
![Page 7: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/7.jpg)
PlatformforOperationalIntelligence
TheSplunkPortfolio
RichEcosystemofApps&Add-Ons
SplunkPremiumSolutions
MainframeData
RelationalDatabases
MobileForwarders Syslog/TCP IoTDevices
NetworkWireData
Hadoop
![Page 8: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/8.jpg)
Agenda
§ Non-TraditionalDataSources
§ DataEnrichment
§ LevelUponSearchandReportingCommands
§ DataModelsandPivot
§ AdvancedVisualizationsandtheWebFramework
8
![Page 9: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/9.jpg)
WorkshopSetup
![Page 10: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/10.jpg)
Non-TraditionalDataSources
![Page 11: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/11.jpg)
Non-TraditionalDataSources
§ NetworkInputs
§ HTTPEventCollector
§ LogEventAlertAction
§ SplunkAppforStream
§ ScriptedInputs
§ DatabaseInputs
§ SplunkODBCDriver
§ ModularInputs
§ zLinux Forwarder
§ MINT
§ Non-SplunkDatastores
11
![Page 12: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/12.jpg)
TraditionalDataSources§ Captureseventsfromlogfilesinrealtime
§ Runsscriptstogathersystemmetrics,connecttoAPIsanddatabases
§ Listenstosyslog andgathersWindowsevents
§ Universallyindexesanydataformatsoitdoesn’tneedadapters
12
Windows• Registry• Eventlogs• Filesystem• sysinternals
Linux/Unix• Configurations• Syslog• Filesystem• Ps,iostat,top
Virtualization• Hypervisor• GuestOS• GuestApps
Applications• Weblogs• Log4J,JMS,JMX• .NETevents• Codeandscripts
Databases• Configurations• Audit/querylogs• Tables• Schemas
Network• Configurations• syslog• SNMP• netflow
![Page 13: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/13.jpg)
NetworkInputs
§ CollectdataoveranyUDPorTCPport§ Somedevicesonlysenddataoveranetworkport
§ BestPractice:usesyslog-ng orrsyslog§ Offerspersistence§ Categorizesdatabyhost
13
![Page 14: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/14.jpg)
HTTPEventCollector(HEC)
§ CollectdataoverHTTPorHTTPSdirectlytoSplunk§ ApplicationDeveloperfocus– fewlinesofcodeinapp
tosenddata§ HECFeaturesInclude:
§ Token-based,notcredentialbased§ IndexerAcknowledgements– guaranteesdataindexing§ RawandJSONformattedeventpayloads§ SSL,CORS(CrossOrigion access),andNetworkRestrictions
14
![Page 15: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/15.jpg)
LogEventAlertAction
§ UseSplunkalertingtoindexacustomlogevent§ Splunksearchableindexofcustomalertevents
§ ConfigurableFeaturesInclude:§ Host§ Source§ Sourcetype§ Index§ Eventtext– constructtheexactsyntaxofthelogevent,
includinganytext,tokens,orotherinformation
15
![Page 16: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/16.jpg)
TheSplunkAppforStream
WireDataEnhancesthePlatformforOperationalIntelligence
Efficient,Cloud-readyWireDataCollection
SimpleDeploymentSupportsFastTimetoValue
16
![Page 17: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/17.jpg)
Stream=BetterInsightsfor*
SolutionArea ContextualData WireData Enriched View
ApplicationManagement
applicationlogs,monitoringdata,metrics,events
protocolconversationsondatabaseperformance,DNSlookups,clientdata,businesstransactionpaths…
Measureapplicationresponsetimes,deeperinsightsforroot-causediagnostics,tracetxpaths,establishbaselines…
IT Operations applicationlogs,monitoringdata,metrics,events
payloaddataincludingprocesstimes,errors,transactiontraces,ICAlatency,SQLstatements,DNSrecords…
Analyzetrafficvolume,speedandpacketstoidentifyinfrastructureperformanceissues,capacityconstraints,changes;establishbaselines…
17
![Page 18: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/18.jpg)
Stream=BetterInsightsfor*SolutionArea ContextualData WireData Enriched View
Security app+infralogs,monitoringdata,events
protocolidentification,protocolheaders,contentandpayloadinformation,flowrecords
Buildanalyticsandcontextforincidentresponse,threatdetection,monitoringandcompliance
DigitalIntelligence
websiteactivity,clickstreamdata,metrics
browser-levelcustomerinteractions
CustomerExperience – analyzewebsiteandapplicationbottleneckstoimprovecustomerexperienceandonlinerevenues
CustomerSupport(online,callcenter)– fasterrootcauseanalysisandresolutionofcustomerissueswithwebsiteorapps
18
![Page 19: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/19.jpg)
ScriptedInputs
19
§ SenddatatoSplunkviaacustomscript§ Splunkindexesanythingwrittentostdout§ Splunkhandlesscheduling§ Supportsshell,Pythonscripts,WINbatch,PowerShell§ Anyotherutilitythatcanformatandstreamdata
StreamingMode§ Splunkexecutesscriptandindexesstdout
§ Checksforanyrunninginstances
WritetoFileMode§ Splunklaunchesscriptwhichproducesoutputfile,noneedforexternalscheduler
§ Splunkmonitorsoutputfile
![Page 20: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/20.jpg)
UseCasesforScriptedInputs
20
§ Alternativetofile-baseornetwork-basedinputs§ Streamdatafromcommand-linetools,suchasvmstat andiostat§ Pollawebservice,APIordatabaseandprocesstheresults§ Reformatcomplexorbinarydataforeasierparsingintoeventsandfields§ Maintaindatasourceswithsloworresource-intensivestartup
procedures§ Providespecialorcomplexhandlingfortransientorunstableinputs§ Scriptsthatmanagepasswordsandcredentials§ Wrapperscriptsforcommandlineinputsthatcontainspecialcharacters
![Page 21: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/21.jpg)
DatabaseInputs
§ Createvaluewithstructureddata§ Enrichsearchresultswithadditionalbusinesscontext
§ Easilyimportdatafordeeperanalysis§ IntegratemultipleDBsconcurrently§ Simpleset-up,non-invasiveandsecure
DBConnectprovidesreliable,scalable,real-timeintegrationbetweenSplunkandtraditionalrelationaldatabases
21
![Page 22: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/22.jpg)
ConfigureDatabaseInputs
22
§ DBConnectApp§ Real-time,scalableintegrationwithrelationalDBs§ Browseandnavigateschemasandtablesbeforedataimport§ Reliablescheduledimport§ SeamlessinstallationandUIconfiguration§ Supportsconnectionpoolingandcaching
§ “Tail”tablesorimportentiretables§ Detectandimportnew/updatedrowsusingtimestampsoruniqueIDs
§ SupportsmanyRDBMSflavors§ AWSRDSAurora,AWSRedShift,IBMDB2forLinux,Informix,MemSQL,MSSQL,MySQL,
Oracle,PostgreSQL,SAPSQLAnywhere(akaSybaseSA),SybaseASEandIQ,Teradata
![Page 23: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/23.jpg)
SplunkODBCDriver
23
§ Interactwith,manipulateandvisualizemachinedatainSplunkEnterpriseusingbusinesssoftwaretools
§ LeverageanalyticsfromSplunkalongsideMicrosoftExcel,TableauDesktoporMicrostrategy AnalyticsDesktop
§ Industry-standardconnectivitytoSplunkEnterprise§ Empowersbusinessuserswithdirectandsecureaccesstomachinedata
§ Combinemachinedatawithstructureddataforbetteroperationalcontext
![Page 24: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/24.jpg)
ODBC:HowitWorks
24
![Page 25: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/25.jpg)
ModularInputs
25
§ Createyourowncustominputs§ Scriptedinputwithstructureandintelligence§ FirstclasscitizenintheSplunkmanagementinterface§ AppearsunderSettings>DataInputs
§ Benefitsoversimplescriptedinput§ Instancecontrol:launchasingleormultipleinstances§ Inputvalidation§ Supportmultipleplatforms§ StreamdataastextorXML§ SecureaccesstomodinputscriptsviaRESTendpoints
![Page 26: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/26.jpg)
ExampleModularInputs
26
Twitter§ StreamJSONdatafromaTwittersourcetoSplunkusingTweepy
AmazonS3OnlineStorage§ IndexdatafromtheAmazonS3onlinestoragewebservice
JavaMessagingService(JMS)§ PollmessagequeuesandtopicsthroughJMSMessagingAPI§ Talkstomultipleproviders:MQSeries (Websphere MQ),ActiveMQ,TibcoEMS,HornetQ,RabbitMQ,NativeJMS,WebLogic JMS,SonicMQ
SplunkWindowsInputs§ RetrieveWINeventlogs,registrykeys,perfmon counters
![Page 27: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/27.jpg)
MoreModularInputs
27
![Page 28: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/28.jpg)
zLinux Forwarder
28
§ EasilycollectandindexdataonIBMmainframes
§ Collectapplicationandplatformdata
§ DownloadasnewForwarderdistributionfors390xLinux
![Page 29: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/29.jpg)
ExtendOperationalIntelligencetoMobileApps
29
DeliverBetterPerforming,MoreReliableApps
DeliverReal-TimeOmni-Channel
Analytics
End-to-EndPerformanceandCapacityInsights
![Page 30: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/30.jpg)
MonitorAppUsageandPerformance
• Improveuserretentionbyquicklyidentifyingcrashesandperformanceissues
• Establishwhetherissuesarecausedbyanapporthenetwork(s)
• Correlateapp,OSanddevicetypetodiagnosecrashandnetworkperformanceissues
30
![Page 31: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/31.jpg)
IntegratedAnalyticsPlatformforDiverseDataStoresFull-featured,IntegratedProduct
FastInsightsforEveryone
WorkswithWhatYouHaveToday
Explore Visualize Dashboards
ShareAnalyze
HadoopClusters NoSQLandOtherDataStores
Hadoop ClientLibraries StreamingResourceLibraries
Bi-directionalIntegrationwithHadoop
![Page 32: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/32.jpg)
ConnecttoNoSQLandOtherDataStores
• Buildcustomstreamingresourcelibraries
• SearchandanalyzedatafromotherdatastoresinHunk
• InpartnershipwithleadingNoSQLvendors
• UseinconjunctionwithDBConnectforrelationaldatabaselookups
![Page 33: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/33.jpg)
VirtualIndexes
§ EnablesseamlessuseofalmosttheentireSplunkstackondata
§ AutomaticallyhandlesMapReduce
§ Technologyispatentpending
![Page 34: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/34.jpg)
DataEnrichment
![Page 35: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/35.jpg)
Agenda
§ Tags – categorizeandaddmeaningtodata
§ FieldAliases – simplifysearchandcorrelation
§ CalculatedFields – shortcutcomplex/repetitivecomputations
§ EventTypes – groupcommoneventsandshareknowledge
§ Lookups – augmentdatawithadditionalexternalfields
35
![Page 36: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/36.jpg)
§ Addsinlinemeaning/context/specificitytorawdata
§ Usedtonormalizemetadataorrawdata
§ Simplifiescorrelationofmultipledatasources
§ CreatedinSplunk
§ Transferredfromexternalsources
WhatisDataEnrichment?
36
![Page 37: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/37.jpg)
§ Addmeaning/context/specificitytorawdata
§ Labelsdescribingteam,category,platform,geography
§ Appliedtofield-valuecombination
§ Multipletagscanbeappliedforeachfield-value
§ Casesensitive
Tags
37
![Page 38: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/38.jpg)
CreateTags
38
SHOW
![Page 39: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/39.jpg)
§ Searcheventswithtaginanyfield
§ Searcheventswithtaginaspecificfield
§ Searcheventswithtagusingwildcards
FindtheWebServersTagsinAction
39
tag=webserver
tag::host=webserver
tag=web*
§ Tagthehostaswebserver
§ Tagthesourcetypeasweb
1
2
3
4
5
SHOW
BacktoSlides
![Page 40: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/40.jpg)
§ Normalizefieldlabelstosimplifysearchandcorrelation§ Applymultiplealiasestoasinglefield
§ Example:Username|cs_username |Userà user§ Example:c_ip |client|client_ipà clientip
§ Processedafterfieldextractions+beforelookups
§ Canapplytolookups
§ Aliasesappearalongsideoriginalfields
FieldAliases
40
![Page 41: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/41.jpg)
Re-LabelFieldtoIntuitiveNameCreateFieldAlias
41
1
2
3
SHOW
![Page 42: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/42.jpg)
§ Createfieldaliasofclientip=customer
§ Searcheventsinlast15minutes,findcustomerfield
§ Fieldalias(customer)andoriginalfield(clientip)arebothdisplayed
SearchusinganIntuitiveFieldNameFieldAliasinAction
42
1
3
2
sourcetype=access_combined
SHOW
![Page 43: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/43.jpg)
§ Shortcutforperformingrepetitive/long/complextransformationsusingevalcommand
§ Basedonextractedordiscoveredfieldsonly
§ Donotapplytolookuporgeneratedfields
CalculatedFields
43
![Page 44: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/44.jpg)
ComputeKilobytesfromBytesCreateCalculatedField
44
1
21
2
3
SHOW
![Page 45: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/45.jpg)
§ Createkilobytes=bytes/1024
§ Searcheventsinlast15minutesforkilobytesandbytes
SearchUsingKilobytesinsteadofBytesCalculatedFieldsinAction
45
1
2
sourcetype=access_combined
SHOW
BacktoSlides
![Page 46: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/46.jpg)
§ Classifyandgroupcommonevents
§ Captureandshareknowledge
§ Basedonsearch
§ Useincombinationwithfieldsandtagstodefineeventtopography
EventTypes
46
![Page 47: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/47.jpg)
§ BestPractice:Usepunctfield§ Defaultmetadatafielddescribingeventstructure§ Builtoninterestingcharacters:",;-#$%&+./:=?@\\'|*\n\r\"(){}<>[]^! »§ Canusewildcards
CreateEventTypes
47
event punct
####<Jun3,20145:38:22PMMDT><Notice><WebLogicServer><bea03><asiAdminServer><WrapperStartStopAppMain><>WLSKernel<><><BEA-000360><ServerstartedinRUNNINGmode>
####<_,__::__>_<>_<>_<>_<>_<>_
172.26.34.223- - [01/Jul/2005:12:05:27-0700]"GET/trade/app?action=logoutHTTP/1.1"2002953
..._-_-_[:::_-]_\"_?=_/.\"__
![Page 48: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/48.jpg)
§ Showpunctforsourcetype=access_combined
§ Pickapunct,thenwildcarditafterthetimestamp
§ AddNOTstatus=200
§ Saveas“bad”eventtype+Color:red+Priority:1(shiftreloadinbrowsertoshowcoloring)
ClassifyEventsasKnownBadCreateEventType
48
eventtype=bad
sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200
1
2
3
4
SHOW
BacktoSlides
![Page 49: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/49.jpg)
LookupstoEnrichRawData
LDAPAD
WatchLists
CRM/ERP
CMDB
ExternalDataSources
Insightcomesout
DatagoesinCreateadditionalfieldsfromtherawdatawithalookuptoanexternaldatasource
![Page 50: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/50.jpg)
§ Augmentraweventswithadditionalfields§ Providecontextorsupportingdetails
§ Translatefieldvaluestomoredescriptivedata§ Example:addtextdescriptionsforerrorcodes,IDs§ Example:addcontactdetailstousernamesorIDs§ Example:adddescriptionstoHTTPstatuscodes
§ File-basedorscriptedlookups
Lookups
50
![Page 51: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/51.jpg)
51
1.Upload/createtable
2.Assigntabletolookupobject
3.Maplookuptodataset
Convert a Code into a DescriptionConfigure a Static Lookup
SHOW
![Page 52: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/52.jpg)
§ GetthelookupfromtheSplunkWiki(saveto.csv file)http://wiki.splunk.com/Http_status.csv
§ Lookuptablefiles>Addnew§ Name:http_status.csv (musthave.csv fileextension)§ Upload:<pathto.csv>
§ Verifylookupwascreatedsuccessfully
1.CreateHTTPStatusTable
52
SHOW
| inputlookup http_status.csv
1
2
3
![Page 53: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/53.jpg)
§ Lookupdefinitions>Addnew§ Name:http_status§ Type:File-based§ Lookupfile:http_status.csv
§ Invokethelookupmanually
2.AddLookupDefinition
53
SHOW
1
2
sourcetype=access_combined | lookup http_status status OUTPUT status_description
![Page 54: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/54.jpg)
§ Automaticlookups>Addnew§ Name:http_status (cannothavespaces)§ Lookuptable:http_status§ Applyto:sourcetype=access_combined§ Lookupinputfield:status§ Lookupoutputfield:status_description
§ Verifylookupisinvokedautomatically
3.ConfigureAutomaticLookup
54
SHOW
1
2
sourcetype=access_combinedBacktoSlides
![Page 55: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/55.jpg)
§ Temporallookupsfortime-basedlookups§ Example:IdentifyusersonyournetworkbasedontheirIPaddress
andthetimestampinDHCPlogs
§ Usesearchresultstopopulatealookuptable§ … | outputlookup <tablename|filename>
§ Callanexternalcommandorscript§ Pythonscriptsonly§ Example:DNSlookupforIPßà Host
§ Createalookuptableusingarelationaldatabase§ ReviewmatchesagainstadatabasecolumnorSQLquery
FancyLookups
55
![Page 56: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/56.jpg)
§ CreatingandManagingAlerts(JobInspector)
§ Macros
§ WorkflowActions
MoreDataEnrichment
56
![Page 57: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/57.jpg)
LevelUponSearch&ReportingCommands
![Page 58: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/58.jpg)
Agenda
§ Doingmorewithbasicsearchcommands
§ Advancedsearchcommands
§ Doingmorewithbasicreportingcommands
58
![Page 59: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/59.jpg)
SearchSyntaxComponents
59
![Page 60: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/60.jpg)
AnatomyofaSearch
60
Disk
![Page 61: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/61.jpg)
§ top– limit§ rare– sameoptionsastop§ timechart– parameters§ stats– functions(sum,avg,list,values,sparkline)§ sort– inlineascendingordescending§ addcoltotals§ addtotals
DoingMorewithBasicSearchCommands
61
![Page 62: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/62.jpg)
WorkshopNotesforPresenter
Tip#5:Inthenextsection,aftereachsearch,havetheparticipantssavethesearchasadashboardpanel.Attheend
oftheworkshop,theywillhavealivingdocumentoftheworkshopexercisestoreferencelater.
Acompleteversionofthisdashboardispackagedasanapp.ItisuploadedtotheBoxfolderasaleavebehind.
62
![Page 63: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/63.jpg)
§ Commandshaveparametersorqualifiers
§ topandrarehavesimilarsyntax
§ Eachsearchcommandhasitsownsyntax– showinlinehelp
FindMostandLeastActiveCustomersUsingthetop+rareCommands
... | top limit=20 clientip
... | rare limit=20 clientip
IPswiththemostvisits
IPswiththeleastvisits
SHOW
![Page 64: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/64.jpg)
§ Sortinlinedescendingorascending
64
... | stats count by clientip | sort - count
... | stats count by clientip | sort + count
Numberofrequestsbycustomer- descending
Numberofrequestsbycustomer- ascending
SorttheNumberofCustomerRequestsUsingthesortCommand
SHOW
![Page 65: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/65.jpg)
§ ShowSearchCommandReferenceDocs§ Functionsforeval+where§ Functionsforstats+chartandtimechart
§ Invokeafunction
§ Renameinline
65
... | stats sum(bytes) by clientip | sort - sum(bytes)
... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes
Totalpayloadbycustomer- descending
Totalpayloadbycustomer- ascending
DetermineTotalCustomerPayloadUsingfunctions+renamecommand
SHOW
![Page 66: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/66.jpg)
§ Listallvaluesofafield
§ Listonlydistinctvaluesofafield
66
... | stats values(action) by clientip
... | stats list(action) by clientip
Activitybycustomer
Distinctactionsbycustomer
ObserveCustomerActivityUsingthelist+valuesFunctions
SHOW
![Page 67: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/67.jpg)
§ Showdistinctactionsandcardinalityofeachaction
67
sourcetype=access_combined| stats count(action) as value by clientip, action| eval pair=action + " (" + value + ")"| stats list(pair) as values by clientip
AnalyzeCustomerActivityCombinelist+valuesFunctions
SHOW
![Page 68: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/68.jpg)
§ Addcolumns
§ Sumspecificcolumns
68
... | stats count by clientip, action
2cols:clientip +action
... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes, count as totalevents by clientip | addcoltotals totalbytes, totalevents
Sumtotalbytesandtotaleventscolums
BuildingaTableofCustomerActivityAddColumnsandSumColumns
SHOW
![Page 69: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/69.jpg)
69
... | stats sum(bytes) as totalbytes, sum(other) as totalother by clientip | addtotals fieldname=totalstuff
Foreachrow,addtotalbytes+totalother
Abetterexample:physicalmemory+virtualmemory=
totalmemory
BuildingaTableofCustomerActivitySumAcrossRows
SHOW
![Page 70: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/70.jpg)
70
... | stats sparkline(count) as trendline by clientip
Incontextoflargereventset
... | stats sparkline(count) as trendline sum(bytes) by clientip
Inlineintables
TrendIndividualCustomerActivitySparklinesinAction
SHOW
BacktoSlides
![Page 71: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/71.jpg)
AdvancedSearchCommandsCommand ShortDescription Hints
transaction Groupeventsbyacommonfieldvalue. Convenient,but resourceintensive.cluster Clustersimilareventstogether. Canbeusedon_raworfield.associate Identifiescorrelationsbetweenfields. Calculatesentropybtn fieldvalues.correlate Calculatesthecorrelationbetween
differentfields.Evaluatesrelationshipof allfieldsinaresultset.
contingency Buildsacontingencytablefortwofields. Computesco-occurrence,or%twofieldsexistinsameevents.
anomalies Computesanunexpectednessscoreforanevent.
Computessimilarityofevent(X)toasetofpreviousevents(P).
anomalousvalue Findsandsummarizesirregular,oruncommon,searchresults.
Considers frequencyofoccurrenceornumberofstdev fromthemean
![Page 72: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/72.jpg)
§ Seweventstogether+createsduration+eventcount
§ Sparklinesinlineintables
72
... | transaction JSESSIONID | table JSESSIONID, action, product_id
GroupbyJSESSIONID
ViewCustomerActivitybySessionUsingthetransactionCommand
SHOW
![Page 73: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/73.jpg)
§ Intelligentgroup(createscluster_countandcluster_label)
§ Sparklinesinlineintables
Cluster
73
SHOW
... | cluster showcount=1 | table _raw, cluster_count, cluster_label
BacktoSlides
![Page 74: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/74.jpg)
§ Predictovertime
§ ChartOverlaywithandwithoutstreamstats
§ Mapswithiplocation+geostats
§ Singlevalue
§ Meteredvisualswithgauge
DoingMorewithBasicReportingCommands
74
![Page 75: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/75.jpg)
§ Predictfuturevaluesusinglower/upperbounds– singleandmultipleseries
75
... | timechart count as traffic | predict traffic
PredictWebsiteTrafficUsingthepredictCommand
SHOW
![Page 76: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/76.jpg)
76
sourcetype=access_combined (action=view OR action=purchase)| timechart span=10m count(eval(action="view")) as Viewed,
count(eval(action="purchase")) as Purchased
CompareBrowsingvs.BuyingActivitySimpleChartOverlay
SHOW
![Page 77: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/77.jpg)
77
... | iplocation clientip | geostats count by clientip
CombineIPlookupwithgeomapping
MapCustomerActivity GeographicallyGeolocation inAction
SHOW
![Page 78: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/78.jpg)
78
... | stats count
DisplayaSimpleCountofEventsSingleValueinAction
SHOW
![Page 79: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/79.jpg)
DisplayCountsUsingGaugesSingleValue,RadialandFillerGaugesinAction
79
... | stats count | gauge count 10000 20000 30000 40000 50000
SHOW
BacktoSlides
![Page 80: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/80.jpg)
DataModelandPivot
![Page 81: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/81.jpg)
Agenda
§ Whatisadatamodel?
§ Buildadatamodel
§ PivotInterface
§ Accelerateadatamodel
81
![Page 82: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/82.jpg)
PowerfulAnalyticsAnyoneCanUse
Enablesnon-technicaluserstobuildcomplexreportswithoutthesearchlanguage
Providesmoremeaningfulrepresentationofunderlyingrawmachinedata
Accelerationtechnologydeliversupto1000xfasteranalyticsoverSplunk5
82
Pivot
DataModel
AnalyticsStore
![Page 83: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/83.jpg)
DefineRelationshipsinMachineDataDataModel• Describeshowunderlyingmachinedataisrepresentedandaccessed
• Definesmeaningfulrelationshipsinthedata
• Enablessingleauthoritativeviewofunderlyingrawdata
Hierarchicalobjectviewofunderlyingdata
Addconstraintstofilteroutevents
![Page 84: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/84.jpg)
TransparentAcceleration
• Automaticallycollected– Handlestimingissues,
backfill…• Automaticallymaintained– Usesaccelerationwindow
• Storedontheindexers– Peertothebuckets
• Faulttolerantcollection
Timewindowofdatathatisaccelerated
Checktoenableaccelerationofdatamodel
HighPerformanceAnalyticsStore
![Page 85: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/85.jpg)
Easy-to-UseAnalytics
• Drag-and-dropinterfaceenablesanyusertoanalyzedata
• Createcomplexqueriesandreportswithoutlearningsearchlanguage
• Clicktovisualizeanycharttype;reportsdynamicallyupdatewhenfieldschange
Selectfieldsfromdatamodel
Timewindow
Allcharttypesavailableinthecharttoolbox
Savereporttoshare
Pivot
![Page 86: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/86.jpg)
§ Definesleastcommondenominatorforadatadomain
§ Standardmethodtoparse,categorize,normalizedata
§ Setoffieldnamesandtagsbydomain§ PackagedasaDataModelsinaSplunkApp
§ Domains:security,web,inventory,JVM,performance,networksessions,andmore
§ MinimalsetuptousePivotinterface
CommonInformationModel(CIM)App
86
![Page 87: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/87.jpg)
§ Apps>FindMoreApps>
§ Search:“CommonInformationModel”
§ Installfree
§ Showfieldsforweb+WebDataModel
DownloadCIMApp
87
SHOW
1
2
3
4
BacktoSlides
![Page 88: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/88.jpg)
DataModel&PivotTutorial
http://docs.splunk.com/Documentation/Splunk/latest/PivotTuto
rial/WelcometothePivotTutorial
88
![Page 89: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/89.jpg)
CustomVisualizationsandtheWebFrameworkToolkit
![Page 90: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/90.jpg)
Agenda
§ DeveloperPlatform
§ WebFrameworkToolkit(WFT)
§ RESTAPIandSDKs
§ GetaFlyingStart
90
![Page 91: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/91.jpg)
OptimizingtheAnalyticsProcess
91
Focusonthedata– intuitivetoolstoenabletheanalyst
Nosinglevisualizationexiststohandlealldatasets.
Neverlosesightoftherawdata
SplunkAnalytics
Explore
Context
Visualize
Algorithms
![Page 92: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/92.jpg)
6.0+6.1:Simple,Interactive,andExtensible
92
VISUALIZATIONEXPLORATION
CUSTOMIZABLEFRAMEWORK
POWERFULANALYTICS
PivotDataModels
InteractiveFormsContextualDrilldown
DashboardEditorWebFramework
![Page 93: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/93.jpg)
TheSplunkEnterprisePlatform
Collection
Indexing
SearchProcessingLanguage
CoreFunctions
Inputs,Apps,OtherContent
SDKContent
CoreEngine
UserandDeveloperInterfaces
WebFramework
RESTAPI
![Page 94: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/94.jpg)
What’sPossiblewiththeSplunkEnterprisePlatform?
PowerMobileApps
LogDirectly
ExtractData
CustomerDashboards
IntegrateBITools
IntegratePlatformServices
Developer Platform
![Page 95: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/95.jpg)
PowerfulPlatformforEnterpriseDevelopersDevelopers Can Customize and Extend
RESTAPI
BuildSplunkApps ExtendandIntegrateSplunk
SimpleXML
JavaScript
HTML5
WebFramework
JavaJavaScriptPython
RubyC#PHP
DataModels
SearchExtensibility
ModularInputs
SDKs
![Page 96: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/96.jpg)
SplunkSoftwareforDevelopers
GainApplicationIntelligence
BuildSplunkApps
IntegrateandExtendSplunk
![Page 97: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/97.jpg)
AWealthofSplunk AppsOver1,100appsavailableontheSplunkappssite
APISDKs UI
Server, Storage, Network
Server Virtualization
Operating Systems
Custom Applications
Business Applications Cloud Services
App Performance MonitoringTicketing/ and Other
WebIntelligence
Mobile Applications
Stream
![Page 98: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/98.jpg)
§ Interactive,cut/pasteexamplesfrompopularsourcerepositories:D3,GitHub,jQuery
§ Splunk6.xDashboardExamplesApphttps://apps.splunk.com/app/1603
§ CustomSimpleXML ExtensionsApphttps://apps.splunk.com/app/1772
§ SplunkWebFrameworkToolkitApphttps://apps.splunk.com/app/1613
ExampleAdvancedVisualizations
98
![Page 99: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/99.jpg)
99
http://www.d3js.org
![Page 100: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/100.jpg)
AddaD3BubbleChart
100
1. GotoFindMoreAppsandInstalltheSplunk6.xDashboardExamplesApp
2. EntertheApp3. GotoExamples>CustomVisualizations>
D3BubbleChart4. Copyautodiscover.js (file)+components/bubblechart (dir)
from:$SH/etc/apps/simple_xml_examples/appserver/staticto:$SH/apps/search/appserver/static
5. CopyandpastesimpleXMLtonewdashboard
SHOW
BacktoSlides
![Page 101: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/101.jpg)
Resources
![Page 102: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/102.jpg)
SplunkDocumentation
102
• http://docs.splunk.com• OfficialProductDocs• Wikiandcommunitytopics• Updateddaily• Canbeprintedto.PDF
![Page 103: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/103.jpg)
SplunkAnswers
103
• http://answers.splunk.com• Communitydriven• Splunksupported• Knowledgeexchange• Q&A
![Page 104: Machine Data 101](https://reader031.vdocuments.net/reader031/viewer/2022021507/586fb2cd1a28abe57d8b69ed/html5/thumbnails/104.jpg)
SplunkEducation
104
• RecommendedforUsers– UsingSplunk– Searching&Reporting
• RecommendedforUI/DashboardDevelopers– DevelopingApps
• Instructor-LedCourses– Web– Onsite