egearmand: an erlang gearman daemon

• OTP compliant distributed application

• Full support of the Gearman binary protocol

• Support of the Gearman administrative protocol

• Focus on reliability and scalability

Concurrency

• Each erlang process can be executed concurrently in any available core

• Concurrent processing of requests

supervisor

WorkersReg FunctionsReg JobsQueue Connections

WorkerProxy

WorkerProxy

WorkerProxy

WorkerProxy

WorkerProxyWorkerProxy

WorkerProxy

WorkerProxy

WorkerProxy

WorkerProxy

WorkerProxyClientProxy

Job Queues

• Queues stored in Erlang’s distributed database Mnesia

• Optional persistence support

• Replication: read nodes (in-memory) and write nodes (fallback, disk copies) can be declared

%% @doc%% Specifies if the Gearman queues should be%% persistent between servers restarts.-spec(persistent_queues() -> true | false) .persistent_queues() -> false .

%% @doc%% If using persitent job queues, nodes where%% to store disc copies.-spec(backup_gearmand_nodes() -> [node()]) .backup_gearmand_nodes() -> [node()] .

configuration.erl

Erlang node

mnesia (disc copy)

Erlang node

mnesia (disc copy)

Erlang node

egearmand

mnesia (memory)

Erlang node

mnesia (disc copy)

Erlang node

mnesia (disc copy)

Erlang node

mnesia (disc copy)

Erlang node

mnesia (disc copy)

egearmand

Clustering

• A set of nodes running egearmand acting as a single gearman server

Clustering

Erlang node

egearmand (master) Erlang node

egearmand (slave)

Erlang node

egearmand (slave) gearman client

gearman worker

Failover

• worker failover

• erlang process failover

• erlang node failover

Worker Failover

• tasks being executed can be re-scheduled in the queue

• specified in configuration

%% @doc%% What to do if a worker fails while executing a task.%% If the value is set to reeschedule the task will be%% queued again.%% If the value is set to none, it will be discarded.-spec(on_worker_failure() -> reeschedule | none) .on_worker_failure() -> reeschedule .

configuration.erl

Erlang Processes Failover

• OTP application

• Supervisor restarts failing processes

Erlang Nodes Failover

• Distributed OTP application

• Handles node crashes

• Application re-started in another node (take over)

• Slave nodes not affected

Extensions

• Extensions support

• Bridge between erlang processes and gearman worker/clients

• Extensions hooked in the request dispatcher

• Associated to certain function names

• RabbitMQ extension included

Erlang node

egearmandrabbitmq

rabbitmqextension

gearman worker

gearman client

/egearmand/rabbitmq/declare

/egearmand/rabbitmq/publish

/egearmand/rabbitmq/consume

%% @doc%% We state which messages are we interested to process-spec(connection_hook_for(atom()) -> boolean()) .connection_hook_for(Msg) -> log:debug(["Checking hook for", Msg]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, _Options]} -> true ; _Other -> false end .

%% @doc%% We state which messages are we interested to process-spec(entry_point(atom(), socket()) -> boolean()) .entry_point(Msg, Socket) -> log:debug(["Entry point of the extensions", Msg, Socket]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, Options]} ->

process_queue_creation(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, Options]} ->

process_queue_publish(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, Options]} ->

process_queue_consume(Options, Socket) ; _Other ->

false end .

rabbitmq_extension.erl

Gotchas

• Early stage of development

• Untested performance

• Local connections (process/node) lost in the event of a failure restart

• Difficult configuration (needs recompiling)

• Not release or boot script

git clone git://github.com/antoniogarrote/egearmand-server.git