Copyright  ©  2013  Splunk  Inc.  

Ma:hew  Se=pane  Professional  Services  Manager,  Splunk  

Onboard  Data  into  Splunk,  Correctly  

#splunkconf  

Legal  NoJces  During  the  course  of  this  presentaJon,  we  may  make  forward-­‐looking  statements  regarding  future  events  or  the  expected  performance  of  the  company.  We  cauJon  you  that  such  statements  reflect  our  current  expectaJons  and  esJmates  based  on  factors  currently  known  to  us  and  that  actual  events  or  results  could  differ  materially.  For  important  factors  that  may  cause  actual  results  to  differ  from  those  contained  in  our  forward-­‐looking  statements,  please  review  our  filings  with  the  SEC.    The  forward-­‐looking  statements  made  in  this  presentaJon  are  being  made  as  of  the  Jme  and  date  of  its  live  presentaJon.    If  reviewed  aUer  its  live  presentaJon,  this  presentaJon  may  not  contain  current  or  accurate  informaJon.      We  do  not  assume  any  obligaJon  to  update  any  forward-­‐looking  statements  we  may  make.    In  addiJon,  any  informaJon  about  our  roadmap  outlines  our  general  product  direcJon  and  is  subject  to  change  at  any  Jme  without  noJce.    It  is  for  informaJonal  purposes  only  and  shall  not,  be  incorporated  into  any  contract  or  other  commitment.    Splunk  undertakes  no  obligaJon  either  to  develop  the  features  or  funcJonality  described  or  to  include  any  such  feature  or  funcJonality  in  a  future  release.  

 

Splunk,  Splunk>,  Splunk  Storm,  Listen  to  Your  Data,  SPL  and  The  Engine  for  Machine  Data  are  trademarks  and  registered  trademarks  of  Splunk  Inc.  in  the  United  States  and  other  countries.  All  other  brand  names,  product  names,  or  trademarks  belong  to  their  respecCve  

owners.    

©2013  Splunk  Inc.  All  rights  reserved.  

2  

About  Me  

!   Professional  Services  Manager  !   More  than  three  years  of  Splunk  experience  !   Involved  in  more  than  300  deployments  from  1GB  to  10TB  

3  

Agenda  

!   Data  !   Splunk  Components  !   Index  Data  !   Proper  Parsing  !   Deploying  in  ProducJon  !   Deployment  Apps  and  Naming  ConvenJons  !   Challenging  Data    

4  

Are  You  in  The  Right  Room  

5  

!   You  have  used  Splunk  at  least  once,  or  at  least  read  about  it  !   You  are  interested  in  Splunk  best  pracJces  !   You  like  to  use  Splunk’s  default  parsing  rules  !   You  just  took  over  a  Splunk  deployment  and  you’re  not  sure    what  to  do  

!   This  is  not  an  educaJon  class;  it’s  best  pracJce  

Data  

6  

!   Machine  data  is  more  than  just  logs  –  it's  configuraJon  data,  data  from  APIs  and  message  queues,  change  events,  the  output  of  diagnosJc  commands  and  more  

!   Log  types:  ApplicaJon,  Web  Access  and  Proxy,  Call  Detail  Records  (CDR),  Clickstream,  Message  Queues,  Packet,  Database  audit  and  tables,  File  audit,  Syslog,  WMI,  PerfMon  

!   Manual:  Ge=ng  Data  In:  h:p://docs.splunk.com/DocumentaJon/Splunk/latest/Data/WhatSplunkcanmonitor  

Splunk  is  the  engine  for  machine  data    

Splunk  Distributed  Components  

7  

Search  Head  

Deployment  Server  

Indexer  

Forwarder  

Test  Environment  

8  

!   Every  Splunk  deployment  should  have  a  test  environment  

!   It  can  be  a  laptop,  virtual  machine  or  spare  server  

!   Should  have  the  same  version  of  Splunk  running  in  producJon  

!   Accessible  to  other  Splunk  developers  and  administrators  

One  Shot  

9  

!   Easiest  way  to  get  data  into  your  test  environment  !   Components  of  the  oneshot:    

 ./splunk add oneshot user_conf.txt –index indexname –soucetype sourcetype name!

!   Where  to  find  more  informaJon:h:p://docs.splunk.com/DocumentaJon/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI  

Data  –  Broken  

10  

Splunk  Apps  

11  

!   Look  to  Splunk  Apps  first  and  uJlize  Technical  Add-­‐On  (TA)  

!   Applies  the  Common  InformaJon  Model  (CIM)    !   CIM  details  the  standard  fields,  event  type  tags,  

and  host  tags  that  Splunk  uses  when  it  processes  most  IT  data  

!   Example  TAs:  –  Windows  –  Unix  –  Exchange  –  AcJve  Directory  –  VMware  Vcenter  –  WebSphere  

Props  

12  

!   Always  set  these  six  parameters      

   # USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

13  

!   Defaults  to  empty      

   # USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

14  

! strpJme  style  format      

  !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

15  

!   By  default  MAX_TIMESTAMP_LOOKAHEAD  =  150  characters      

! !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

16  

!   By  default  set  to  True      

  !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

17  

!   By  default  set  to  ([\r\n]+);  change  to  posiJve  lookahead      

! !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

18  

!   By  default  set  to  10000  bytes;  set  to  0  to  never  truncate      

! !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Props  

19  

!   By  default  set  to  10000  bytes;  set  to  0  to  never  truncate      

! !# USER CONFERENCE!

! ![user_conf_2012]!

! !TIME_PREFIX = ^!

! !TIME_FORMAT = %Y-%m-%d %H:%M:%S!

! !MAX_TIMESTAMP_LOOKAHEAD = 19!

! !SHOULD_LINEMERGE = False!

! !LINE_BREAKER = ([\n\r]+)(?=\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!

! !TRUNCATE = 999999!

Duplicate  slide?  

Data  –  Fixed  

20  

ProducJon  Deployment  

ProducJon  Environment  

22  

!   Complexity  managing  configuraJons  across  tens,  hundreds,  or  thousands    of  forwarders  

!   Not  all  indexers  and    search  heads  receive  the    same  configuraJons  

!   Should  think  about  version  control  for  deployment  apps,  e.g.,  GitHub  

SHP  

Deployment  Server  Terminology  

23  

!   Deployment  Server  –  A  Splunk  instance  that  acts  as  a  centralized  configuraJon  manager,  grouping  together  and  collecJvely  managing  any  number  of  Splunk  instances.  Any  Splunk  instance  can  act  as  a  deployment  server,  even  one  that  is  indexing  data  locally.  Splunk  instances  that  are  remotely  configured  by  deployment  servers  are  called  deployment  clients  

!   Deployment  Client  –  A  Splunk  instance  that  is  remotely  configured  by  a  deployment  server  

!   Server  Class  –  Represents  a  configuraJon  of  Splunk  deployment  clients.  Server  classes  enable  the  management  of  a  group  of  deployment  clients  as  a  single  unit.  A  server  class  can  be  used  to  group  deployment  clients  together  by  applicaJon,  OS,  data  type  to  be  indexed,  or  any  other  feature  of  your  Splunk  deployment  

Deployment  App  

24  

!   A  deployment  app  (configuraJon  bundle)  is  a  set  of  deployment  content  (including  configuraJon  files)  deployed  as  a  unit  to  clients  of  a  server  class  

!   Located  in  $SPLUNK_HOME/etc/deployment-­‐apps  and  pushed  to  deployment  client’s  $SPLUNK_HOME/etc/apps  folder  

! DO  NOT  store  configuraJons  in  $SPLUNK_HOME/etc/system/local  

!   Use  deployment  apps  regardless  of  your  deployment  tool    

Deployment  App  –  Naming  ConvenJon  

25  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

Base  

inputs  

Deployment  App  –  Naming  ConvenJon  

26  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

base  

inputs  

Deployment  App  –  Naming  ConvenJon  

27  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

base  

inputs  

Deployment  App  –  Naming  ConvenJon  

28  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

base  

inputs  

Deployment  App  –  Naming  ConvenJon  

29  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

base  

inputs  

Deployment  App  –  Naming  ConvenJon  

30  

org  

acme  

acme  

splk  

splk  

group  

finance  

markeJng  

all  

ps  

applicaJon  

apache  

iis    

indexer  

user_conf  

configuraJon  

inputs  

props  

base  

inputs  

splk_ps_user_conf_inputs  

Deployment  Apps  

31  

! SplunkForwarder  

! SplunkLightForwarder  

! Splunk_for_AcJveDirectory  

! Splunk_for_Exchange  

! splk_all_deploymentclient  

! splk_all_forwarder_outputs  

! splk_all_indexer_base  

! splk_all_search_base  

! splk_ps_user_conf_inputs  

! splk_ps_user_conf_props  

! splk_ps_user_conf_web  

! splunk_app_was  

!   user-­‐prefs  

msettipane-mba13:apps msettipane$ ls -la!

Challenging  Data  

Limit  Indexed  Data  

33  

! Anonymize  data:    [source::.../accounts.log]!

!SEDCMD-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!

 

!   Rewrite  raw  data:    [source::.../sql.log]!

!SEDCMD-sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g!

 

!   Discard  events:  !props!

![source::/var/log/user_conf.txt]!

!TRANSFORMS-null= setnull!

transforms![setnull]!REGEX =!(?i)DEBUG!DEST_KEY = !queue!FORMAT = !nullQueue!

Limit  Indexed  Data  

34  

! Anonymize  data:  ![source::.../accounts.log]!

!SEDCMD-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!

 

!   Rewrite  raw  data:  ![source::.../sql.log]!

!SEDCMD-sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g!

 

!   Discard  events:  !props!

![source::/var/log/user_conf.txt]!

!TRANSFORMS-null= setnull!

transforms![setnull]!REGEX =!(?i)DEBUG!DEST_KEY = !queue!FORMAT = !nullQueue!

Limit  Indexed  Data  

35  

! Anonymize  data:  ![source::.../accounts.log]!

!SEDCMD-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!

 

!   Rewrite  raw  data:  ![source::.../sql.log]!

!SEDCMD-sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g!

 

!   Discard  events:  !props!

![source::/var/log/user_conf.txt]!

!TRANSFORMS-null= setnull!

transforms![setnull]!REGEX =!(?i)DEBUG!DEST_KEY = !queue!FORMAT = !nullQueue!

CollecJng  Syslog  

36  

!   Send  device,  e.g.,  routers,  firewalls  to  a  syslog  collector  

!   Write  files  to  this  directory  structure:  /sourcetype/host/log.txt  

!   Monitor  the  sourcetype  level   cisco_asa  

my.firewall.name  # CISCO ASA![monitor:///data/cisco_asa/…/]!sourcetype = cisco_asa!host_segment = 3!index = firewall!!

Check  for  Header  

37  

!   Steps  to  fixing  sourcetype-­‐2,  3,  4  problems  (e.g.,  iis-­‐2,  iis-­‐3)  !   Address  issue  on  forwarder:  CHECK_FOR_HEADER  =  False  !   Extract  fields  using  delimiter:  

 [sourcetype]!DELIM = “,”!FIELDS = “one”, “two”, three”!

!   On  search  head  rename  already  indexed  events:  rename  =  iis  

MulJple  Timestamps  

38  

dateLme.xml  <datetime>!

<define name=”two_tz" extract="day, litmonth, year, hour, minute, second, zone">!

  <text><![CDATA[^(\d+)-(\w+)-(\d+),(\d+):(\d+):(\d+),(?:[^,]*,){2}([\w\-]*)]]></text>! </define>!

<timePatterns> !

  <use name=”two_tz">!

</timePatterns>!

<datePatterns> !

  <use name=”two_tz">!

</datePatterns>!

</datetime>!

 

 

props.conf  # USER CONF!

[user_conf]!

DATETIME_CONFIG = /etc/apps/splk_ps_user_conf_props/local/datetime.xml!

*  Do  not  set  TIME_FORMAT  

12-­‐Sep-­‐2012,09:01:00,12-­‐Sep-­‐2012,09:02:00,-­‐4  INFO    Jtle="User  Conference"  msg="Splunk  hosted  user  conference  in  Las  Vegas."  12-­‐Sep-­‐2012,19:01:00,12-­‐Sep-­‐2012,19:02:00,-­‐5  DEBUG  Jtle="User  Conference"  msg="Ge=ng  Data  In,  Correctly  is  a  solid  session."  

Summary  

39  

!   Test  in  a  non-­‐producJon  environment  !   Always  use  key  props  parameters:    

–  TIME_PREFIX  –  TIME_FORMAT  –  MAX_TIMESTAMP_LOOKAHEAD  –  SHOULD_LINEMERGE  –  LINE_BREAKER  –  TRUNCATE  

!   Deploy  apps  to  /etc/apps;  not  /etc/system/local  !   Clear  predictable  naming  convenJon  !   When  you’re  stuck,  use  Splunk  Answers    

Resources  

40  

!   Get  educated:  h:p://www.splunk.com/view/educaJon/SP-­‐CAAAAH9  

!   Download  Splunk  applicaJons:  h:p://splunk-­‐base.splunk.com/apps/  

!   Hire  Splunk  Professional  Services:  h:p://www.splunk.com/view/professional-­‐services/SP-­‐CAAABH9  

!   Watch  some  videos:  h:p://www.splunk.com/videos  

Next  Steps  

41  

Download  the  .conf2013  Mobile  App  If  not  iPhone,  iPad  or  Android,  use  the  Web  App    

Take  the  survey  &  WIN  A  PASS  FOR  .CONF2014…  Or  one  of  these  bags!    Go  to  the  Search  Party!  Marquee  Nightclub  at  The  Cosmopolitan  Today,  7:30-­‐10:30pm  

1  

2  

3  

THANK  YOU