pivotal open source: using fluentd to gain insights into your logs
TRANSCRIPT
Fluentd:Towards Unified Logging Layer
Kiyoto Tamura
@kiyototamura
Treasure Data, Inc.
• Fluentd maintainer &
community manager
• data nerd
• math nerd
• nerd
whoami
this talk isn’t about
Big Data
it’s about
Log Data
a motivating anecdote
The life of
a data scientist
(me in 2009)
http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext
Acquire Data (or so you think)
WUT!? Invalid UTF8?
Fix the encoding issue…
Yell at the engineers
Some columns are missing!?
Run the script…DIVISION
BY ZERO!!!
Hmm…
Logging.priority
=> :not_super_high
analytics.priority
=> :very_high
analytics.needs? :logs
=> true
outage.priority
=> :super_high
outage.needs? :logs
=> [“no”, “shit”]
Metrics and Monitoring
(hint: you need logs)
Ops
VPs
Engineers
Managers
PMs
More PMs
How can we do better?
How to Unify Logging (1)Common Interface + Decoupling
WebMobile IoT
Message Queue
Search Backend
Analytic DBArchival Storage
Unified Logging Layer
Parse into acommon data format
Decouple fromdata sources
How to Unify Logging (2)Reliability & Scalability
WebMobile IoT
Message Queue
Search Backend
Analytic DBArchival Storage
Unified Logging Layer
Need persistence/buffering
Robust retriesand recovery
How to Unify Logging (3)Extensibility
Web? IoT
? Search Backend
Analytic DBArchival Storage
Unified Logging Layer
Adding a new in/output must be easy
Same for filters
Fluentd can help us unify logging
how it works
127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}
Parse as JSON!
?
[“05/Feb/2012:17:11:55”,“web.access”,{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}]
timestamp tag
record
?
web.mongodb
web.file
web.hdfs
web.s3
web.mysql
Demo: Bring Your Own A/B Testing
How A/B Testing Starts
website
A/B Testing SaaS
<script>…</script>
How A/B Testing Evolves
A/B Testing SaaS 1
<script>…</script>
website
A/B Testing SaaS 1
A/B Testing SaaS 1
A/B Testing SaaS 1
<script>…</script>
event.post()…
<script>…</script>
Android iOS
event.post()…
How A/B Testing Evolves
A/B Testing SaaS 1
<script>…</script>
website
A/B Testing SaaS 1
A/B Testing SaaS 1
A/B Testing SaaS 1
<script>…</script>
event.post()…
<script>…</script>
Android iOS
event.post()…
Looks Familiar?
Bring Your Own A/B Testing!
websiteAndroid iOS
A/B Testing SaaS 1
A/B Testing SaaS 2
Analytic DBArchival Storage
bit.ly/cf-fluentd
{
“install”: “gem install fluentd”,
“website”: “www.fluentd.org”,
“github” : “fluent/fluentd”,
“twitter”: “@fluentd”
}