fluent event collector [update]
TRANSCRIPT
![Page 1: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/1.jpg)
Sadayuki Furuhashi
Fluent
@frsyuki
e Event Collector Service
Treasure Data, Inc.
Structured logging
Pluggable architecture
Reliable forwarding
![Page 2: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/2.jpg)
What’s Fluent?
It's like syslogd, but uses JSON for log messages
![Page 3: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/3.jpg)
What’s Fluent?
“2011-04-01 host1 myapp: cmessage size=12MB user=me”
2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}
![Page 4: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/4.jpg)
What’s Fluent?
2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}
time tag
record
![Page 5: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/5.jpg)
Fluent vs Scribe
• Deals with structured logs• Easy to install> “gem install fluentd”> also apt-get and yum http://packages.treasure-data.com/
• Easy to customize> add/modify plugins without compile> “gem search -rd fluent-plugin”
![Page 6: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/6.jpg)
Fluent vs Flume
• Easy to setup> “sudo fluentd --setup && fluentd”
• Compact> small engine (2,000 lines) + plugins> JVM-free
• Easy to configure
![Page 7: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/7.jpg)
# receive events via HTTP<source> type http port 8888</source>
# read logs from file<source> type tail path /var/log/httpd.log format apache tag apache.access</source>
# save alerts to file<match alert.**> type file path /var/log/fluent/alerts</match>
# save access logs to MongoDB# and forward to other server<match apache.access> type copy <store> type mongo host 127.0.0.1 </store> <store> type forward <server> host 192.168.0.11 weight 20 </server> <server> host 192.168.0.12 weight 60 </server> </store></match>
![Page 8: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/8.jpg)
Architecture
Input Buffer Output
HTTP+JSONFile tailSyslog...
MemoryFile
FileAmazon S3Fluent...
Pluggable Pluggable Pluggable
![Page 9: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/9.jpg)
Architecture
Input
HTTP+JSONFile tailSyslog...
Pluggable
✓ Receive logs✓ Or pull logs from data sources✓ Non-blocking
Input plugins:
![Page 10: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/10.jpg)
Architecture
Pluggable
✓ Improve performance✓ Provide reliability✓ Provide thread-safety
Buffer plugins:
Buffer
MemoryFile
![Page 11: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/11.jpg)
Architecture
Pluggable
✓ Improve performance✓ Provide reliability✓ Provide thread-safety
Buffer plugins:
chunk
chunk
chunk output
Input
![Page 12: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/12.jpg)
Architecture
Pluggable
✓ Write or send event logs
Output plugins:
Output
FileAmazon S3Fluent...
![Page 13: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/13.jpg)
Log collection
FluentApps
Client library
Apache log
HTTP+JSON(in_http)
TCP+MessagePack(in_tcp)
follow growth of files (in_tail)
![Page 14: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/14.jpg)
Client library
Client library
Ruby (by @frsyuki)Python (by @kzk_mover)PHP (by @chobi_e)Java (by @muga_nishizawa)C++ (by @nobu_k)...
Fluent.open(“myapp”, “localhost”)Fluent.event(“login”, {“user”=>38})#=> 2011-02-03 04:56:01 myapp.login {“user”:38}
![Page 15: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/15.jpg)
Forwarding
Fluent
FluentApps
FluentApps
FluentApps
![Page 16: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/16.jpg)
HA Forwarding
Fluent
Fluent
FluentApps
FluentApps
FluentApps
Active
Active
✓ load balancing✓ continuous heartbeat✓φ accrual failure detector
![Page 17: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/17.jpg)
External filter program
FluentFluent
Externalprogram
・stdin / stdout・TSV
![Page 18: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/18.jpg)
External output program
FluentFluent
Externalprogram
Externalprogram
・TSV file
![Page 19: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/19.jpg)
External input program
Fluent
Externalprogram
Externalprogram
Externalprogram
receiveor pull
・stdout・TSV
![Page 20: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/20.jpg)
Incremental Aggregationby @doryokujin #fluent_meetup2
FluentApps MongoDB
/ Redis
count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/landing”] += 1count[“2011-01-01 02h”.”/index”] += 1...
count[“2011-01-01 01h”.”/index”] == 42count[“2011-01-01 01h”.”/landing”] == 38count[“2011-01-01 02h”.”/index”] == 55count[“2011-01-01 02h”.”/landing”] == 55
FluentApps
![Page 21: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/21.jpg)
MySQL replicator (plan)
FluentMySQL
read binlog continuouslyusing MySQL binlog API
mysql://127.0.0.1
http://www.slideshare.net/mkindahl/binary-log-api-presentation-oscon-2011
![Page 22: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/22.jpg)
3rd Party Plugins
• Scribe input/output (by @kzk_mover)> receive logs sent from scribed
• Amazon S3 output (by @frsyuki)> upload log files to Amazon S3
• MongoDB output (by @repeatedly)> write structured logs to MongoDB
• Redis output (by @yuki24)> write structured logs to Redis
![Page 23: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/23.jpg)
3rd Party Plugins
• Cassandra output (by @railute)> write structured logs to Cassandra
• Growl output (by @goth_writst_cut)> notify messages using Growl(Growl: notification service for Mac OS X)
• Aggregation (by @doryokujin)> incremental aggregation using NoSQL DBs
• ...
![Page 24: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/24.jpg)
http://fluentd.org/plugin/> gem search -dr fluent-plugin
![Page 25: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/25.jpg)
Plugin Developer APIs
• Unit test framework (like “MRUnit”)> Fluent::Test::InputTestDriver> Fluent::Test::OutputTestDriver> Fluent::Test::BufferedOutputTestDriver
• Fluent::BasicTail (base class of “tail” plugin)> text parser is customizable def parse_line(line)
![Page 26: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/26.jpg)
What’s Fluent?
• Event collector service> deals with structured event logs
• Compact> small engine (2,000 lines) + plugins
• Plugin archtecture> install plugins using RubyGems
![Page 28: Fluent event collector [update]](https://reader033.vdocuments.net/reader033/viewer/2022042521/543fc353b1af9f4e0a8b4a5c/html5/thumbnails/28.jpg)
Demo