low latency logging with rabbitmq (php london - 4th sep 2014)
DESCRIPTION
Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come up. We’re going to examine a tried and tested method to carry out high-performance, low-latency logging using the power of RabbitMQ to ensure minimal impact to the performance of your runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.TRANSCRIPT
Low Latency Loggingwith RabbitMQ
PHP London4th September 2014
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Who is this guy?
Let’s go back to basics...
Errors
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
error_reporting(0);
They look rubbish!
@
Log to file only
Ways Around// Will raise E_WARNING
fopen($somefile, 'r');
Ways Around// No error! :)
if (file_exists($somefile)) {
fopen($somefile, 'r');
} else {
// nice handling...
// maybe throw exception...
}
Exceptions
Jargon Buster
● throwTriggering
● tryTry to run
● catchHandle it
● finallyRun code after try/catch
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
C♯/C++
Catchable
Turn into fatal errorsif not caught
Classy
Descriptive exceptions
Example (exception class)class DivisionByZeroException
extends LogicException
{
}
Example (throw)class Mathematics {
public function divide($a, $b) {
if ($b == 0) {
$msg = sprintf(“Divide %s by zero”, $a);
throw new DivisionByZeroException($msg);
}
return ($a / $b);
}
}
Example (catch)$maths = new Mathematics();
try
{
$result = $maths->divide(5, 0);
}
catch (DivisionByZeroException $exception)
{
$logger->warning($exception->getMessage());
}
Logging
What is “logging”?
What is “logging”?Keeping a record of all events...
What is “logging”?Keeping a record of all events...
exceptions, errors, warnings, info, debug
Logging is a paper trail
Log like you just don’t care
Log wherever you like
What about Apache’s error_log?
source: http://up-ship.com/blog/?p=20903
Doin’ it rightwrong… // Magic file that makes your entire project work perfectly
@ini_set('display_errors', 0);
@error_reporting(0);
function __globalErrorHandler() {
return true;
}
@set_error_handler('__globalErrorHandler');
@set_exception_handler('__globalErrorHandler');
@register_shutdown_function(function() {
if(error_get_last())
echo "Script executed successfully!";
}); https://github.com/webarto/boostrap.php
Requirements (for everyone)
● Fire & forget● Minimum or zero latency● Highly available● Log everything● PSR-3 compatible
PSR-3
● monolog (PSR-3)● Drupal - PSR-3 Watchdog● phpconsole● log4php● RavenPHP + Sentry● FirePHP (dev environment)● Roll your own
Logging Options
source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
Capturing Logging
Use “capture methods”, send to $logger
● set_exception_handler()○ Handles all uncaught exceptions
● set_error_handler()○ Handles most errors
● register_shutdown_function()○ Handles fatal errors
Sending Log Messages
● Handler/Adapter translates● However you want…● Monolog has loads:
○ syslog-compatible / error_log○ Email, HipChat○ AMQP, Sentry, Zend Monitor, Graylog2○ Redis, MongoDB, CouchDB
Capture Method
Logger (PSR-3)
Handler / Adapter
Data Storage
Typical PSR-3 Compatible Design
Monolog\ErrorHandler->handleException()
Monolog\Logger->log()
Monolog\Handler->handle()
Monolog
Low LatencyHigh Performance
Slow Logging
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
Acknowledge message
HTTP response to client
Zero Latency Logging (ideal)
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
HTTP response to client
UDP?
Low Latency Logging (balance)
ApplicationBrowser Log Server
HTTP request
Send log message to database
Error!
HTTP response to client
…so how?
Say hello to RabbitMQ
RabbitMQ - Basic
source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
Producer Consumer
test_queue
1 2 3 4 5
RabbitMQ - Exchanges
source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html
Exchange
Consumertest_queue
1 2 3 4 5
Consumer
Consumer
Consumer
Consumer
Using Queues === Fast!Add RabbitMQ to logging architecture...
Ed\Log\Handler\ErrorHandler->handleException()
Ed\Log\Logger->log()
Ed\Log\Publisher\AmqpPublisher->publish()
Logging Server
Low Latency (using AMQP)
RabbitMQ JSON payload
Fetch message
Low Latency Logging (with AMQP)
ApplicationBrowser Log Server
HTTP request
JSON via AMQP
Error!
HTTP response
RabbitMQ
Why bother?● Scalability
RabbitMQ
Application A
Application B
Application C
Log Worker
Log Worker
Log Worker
Log Worker
Single Point of Failure...
● RabbitMQ can do HA or clustering
RabbitMQNode 1
RabbitMQNode 3
RabbitMQNode 2
RabbitMQNode 4
RabbitMQNode 5
RabbitMQNode 6
Questions?
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Thanks for watching!