splunk's api how we built it
DESCRIPTION
These are the slides for my September API Craft SF talk on how we built / design Splunk's API. You can find the video here: https://www.youtube.com/watch?v=vHXcDKK4eGY. My talk starts at about 60 mins. The other two are on Uber and Sift Science and you should watch them as well!TRANSCRIPT
![Page 1: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
![Page 2: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/2.jpg)
Agenda
Why we built Splunk’s API
How we built Splunk’s API
![Page 3: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/3.jpg)
What is Splunk
A product for handling REALLY large and varied sets of evented data
![Page 4: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/4.jpg)
What is Splunk
From megabytes to hundreds of terabytes daily
![Page 5: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/5.jpg)
What is Splunk
It is highly scalable and distributed
![Page 6: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/6.jpg)
What is Splunk
Useful in many domains: IT/Ops/DevOps, security, healthcare, financial,IOT/Devices
![Page 7: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/7.jpg)
7
Splunk storage Other Big Data stores
Developer Pla6orm
Data collecUon and indexing
Report and analyze
Custom dashboards
Monitor and alert
Ad hoc search
![Page 8: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/8.jpg)
Why – Product need
Need to build a Splunk UI that surfaces all of Splunk’s capabiliUes
![Page 9: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/9.jpg)
Why – Customer need
Need to allow customers to integrate their applicaUons and scripts with Splunk
![Page 10: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/10.jpg)
Why – Cost
Less code to maintain, a single source of truth
![Page 11: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/11.jpg)
Why – Reach
HTTP is ubiquitous, every plaZorm has a client
![Page 12: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/12.jpg)
The arch
12
![Page 13: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/13.jpg)
Server Tech Stack
13
C/C++
Python – Cherry PI Python – Django
Javascript Node
![Page 14: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/14.jpg)
Client Tech Stack
14
Javascript Node Python Ruby PHP C#
![Page 15: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/15.jpg)
The API
15
![Page 16: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/16.jpg)
Log directly to Splunk via HTTP
Run historical and real-‐Ume searches
What can you do with Splunk’s API?
16
Search Manage
Add/Delete Users
ReporUng/Alerts
Manage Inputs
ConfiguraUon
Index
Login to a Splunk instance and get a session token
Auth
![Page 17: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/17.jpg)
The API design
17
Service
Categories
Endpoints
Endpoints
![Page 18: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/18.jpg)
The API design -‐ Categories
18
![Page 19: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/19.jpg)
The API design – Endpoints
19
![Page 20: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/20.jpg)
The API design – Endpoints
20
![Page 21: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/21.jpg)
Responses -‐ Feeds and Hypermedia <entry xmlns="hjp://www.w3.org/2005/Atom" xmlns:s="hjp://dev.splunk.com/ns/rest" xmlns:opensearch="hjp://a9.com/-‐/spec/opensearch/1.1/"> <Utle>search index</Utle> <id>hjps://localhost:8089/services/search/jobs/mysearch_02151949</id> <updated>2011-‐07-‐07T20:49:58.000-‐07:00</updated> <link href="/services/search/jobs/mysearch_02151949" rel="alternate"/> <published>2011-‐07-‐07T20:49:57.000-‐07:00</published> <link href="/services/search/jobs/mysearch_02151949/search.log" rel="search.log"/> <link href="/services/search/jobs/mysearch_02151949/events" rel="events"/> <link href="/services/search/jobs/mysearch_02151949/results" rel="results"/> <link href="/services/search/jobs/mysearch_02151949/results_preview" rel="results_preview"/> <link href="/services/search/jobs/mysearch_02151949/Umeline" rel="Umeline"/> <link href="/services/search/jobs/mysearch_02151949/summary" rel="summary"/> <link href="/services/search/jobs/mysearch_02151949/control" rel="control"/> </entry>
21
![Page 22: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/22.jpg)
Auth
22
HTTP Basic Token based LDAP/AD
Cookie based
![Page 23: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/23.jpg)
Auth – HTTP Basic
23
curl -‐k -‐u admin:changeme hjps://localhost:8089/services/auth/login -‐d username="admin" -‐d password="changeme"
![Page 24: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/24.jpg)
Auth – Splunk Token
24
curl -‐k -‐H "AuthorizaUon: Splunk SfH2D^zvPyLu^mO61C9kWtB7TOuQs0i9oSzh4lD7ho7Gvw26I61VYRjXkgjLQlJDJ0hER^q^A6v0BHYiKNba^CMbOmC63frGCrDqr2Zt" hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get
![Page 25: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/25.jpg)
Search – Oneshot – Get me results!
25
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot"
![Page 26: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/26.jpg)
Search – Oneshot – Get me results in json
26
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json"
![Page 27: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/27.jpg)
Search – Oneshot – Get me json columns
27
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
![Page 28: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/28.jpg)
Search – Oneshot – Get me json rows
28
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
![Page 29: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/29.jpg)
Search – Blocking – Wait Ull done!
29
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5” -‐d exec_mode=”blocking” output_mode="json” | python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
![Page 30: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/30.jpg)
Search – List search jobs
30
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get | python -‐mjson.tool
![Page 31: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/31.jpg)
Search – Normal– Run in the background
31
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 10000" -‐d exec_mode="normal" output_mode="json"| python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
![Page 32: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/32.jpg)
Search -‐ Export
32
curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal | head 100000" -‐d output_mode="raw"
![Page 33: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/33.jpg)
Search – Export REALTIME
33
curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal" -‐d output_mode="raw" earliest_Ume="rt-‐1m" latest_Ume="rt"
![Page 34: Splunk's api how we built it](https://reader034.vdocuments.net/reader034/viewer/2022052301/558c7a5fd8b42a45318b45f7/html5/thumbnails/34.jpg)
Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
dev.splunk.com splunk.com/jobs