Rusty Klophaus - @rustyioBasho Technologies

EVERYBODY POLYGLOT!(Cross-Language RPC with Erlang)

ErlangDC · December 2011

@rustyio

Languages have Strengths

2

http://wordaligned.org/articles/distorted-software

@rustyio

Connections are HardSerialization

Versioning & UpgradesData Type Mismatches

Speed, Bottlenecks & Back-PressureInadequate ToolingContext Switching

3

@rustyio

Connections are Hard

4

http://wordaligned.org/articles/distorted-software

@rustyio

This Talk

5

Create An Example ServiceAny service will do, just need a framework for discussion.

Expose Application via Interfaces• REST / JSON via Webmachine & Spooky• Protocol Buffers via erlang-protobuffs• BERT-RPC via Ernie Server

Codehttp://github.com/rustyio/ErlangRPCDemo

@rustyio

This Talk

6

Our Application: A Sequence ServerErlang service that returns a sequence of numbers.

1 sequence(N) -> 2 List1 = lists:seq(1, N), 3 List2 = [list_to_binary(integer_to_list(X)) || X <- List1], 4 {ok, List2}; 5 6 %% sequence(2). 7 {ok, [<<"1">>, <<"2">>]}. 8 9 %% sequence(5). 10 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, <<"5">>]}. 11 12 %% sequence(50000). 13 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, ...]}.

@rustyio

OverviewREST?

Protocol Buffers?BERT-RPC?

7

@rustyio

REST / JSON - Overview

8

REST - Representational State TransferConvention for talking to applications over HTTPActions are Verbs are HTTP Methods (GET/PUT/POST/DELETE/...)

Objects are nouns are URLs

JSON - Javascript Object NotationEncode data as parseable JavascriptUnderstood by everythingHuman Readable

GET /users/5

{"id":5,"first":"Rusty","last":"Klophaus"}

@rustyio

REST / JSON - Strengths & Weaknesses

9

StrengthsSimple, easy to poke aroundGood support in every languageComposable - Caches, Reverse Proxies, Load Balancers

WeaknessesGeneral == More Handshaking/Metadata == More Overhead

@rustyio

Protocol Buffers - Overview

10

Protocol BuffersDeveloped by Google It’s not a protocol, it’s a format:• You provide the client / server logic • Useful for transmission AND storageDefine data structures in .proto file, generate code

http://code.google.com/apis/protocolbuffers/https://github.com/ngerakines/erlang_protobuffs

@rustyio

Protocol Buffers - Strengths & Weaknesses

11

StrengthsCompactAdd fields without breaking existing applications• Versioning is not strict

WeaknessesConfiguration file (.proto) with new syntax to learnGenerated codeUneven language support

@rustyio

BERT-RPCDeveloped by GitHub (Tom Preston-Werner) BERT = Binary Erlang Term• Encoding mimics native Erlang serializationhttp://bert-rpc.orghttps://github.com/mojombo/ernie

BERT-RPC - Overview

12

1 % Request 2 {call, ernie_sequence, sequence, [3]} 3 4 % Response 5 {response, {ok, [<<"1">>, <<"2">>, <<"3">>]}}

@rustyio

BERT-RPC - Strengths & Weaknesses

13

StrengthsEasy to set upAgileCompact

WeaknessesUneven language supportLess buzz than it deservesNot fully product-ized

@rustyio

Requests & ResponsesWhat does the chatter look like?

14

@rustyio

REST / JSON - Request & Response

15

GET /sequence/3 HTTP/1.1User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5Host: localhost:8001Accept: */*

HTTP/1.1 200 OKServer: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)Date: Wed, 23 Nov 2011 22:09:13 GMTContent-Type: application/jsonContent-Length: 21

["1","2","3"]

Request

Response

@rustyio

Protocol Buffers - Encodingrpc_demo.proto

16

1 message SequenceRequest { 2 required uint32 n = 1; 3 } 4 5 message SequenceResponse { 6 repeated bytes sequence = 1; 7 }

@rustyio

Protocol Buffers - RPC

17

1 rpc_demo_pb:encode({sequencerequest, 3}). 2 <<8,3>>

1 rpc_demo_pb:encode({sequenceresponse, [<<"1">>, <<"2">>, <<"3">>]}). 2 <<10,1,49,10,1,50,10,1,51>>

Request

Response

@rustyio

BERT-RPC - Encoding

18

1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]). 2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 3 4 5 bert:encode([<<"1">>,<<"2">>,<<"3">>]). 6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>

@rustyio

BERT-RPC - Encoding

19

1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]). 2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 3 4 5 bert:encode([<<"1">>,<<"2">>,<<"3">>]). 6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 7 | | | | | | 8 | | | | | + Ascii value for '1' 9 | | | | + Length of string (1) 10 | | | + Next term is a string 11 | | + Length of list (3) 12 | + Next term is a list 13 + Start of Erlang term

@rustyio

BERT-RPC - RPC

20

1 bert:encode({call, ernie_sequence, sequence, [5]}).

3 <<131,104,4,100,0,4,99,97,108,108,100,0,14,101,114,110, 4 105,101,95,115,101,113,117,101,110,99,101,100,0,8,115, 5 101,113,117,101,110,99,101,107,0,1,5>>

1 bert:encode({reply,{ok,[<<"1">>,<<"2">>,<<"3">>,<<"4">>,<<"5">>]}}). 2 3 <<131,104,2,100,0,5,114,101,112,108,121,104,2,100,0,2,111, 4 107,108,0,0,0,5,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0, 5 1,51,109,0,0,0,1,52,109,0,0,0,1,53,106>>

Request

Response

@rustyio

Show me the Client Code!

21

@rustyio

REST / JSON - Clients

22

curl http://localhost:8001/sequence/5

CURL

1 url = "http://localhost:8001/sequence/5" 2 resp = Net::HTTP.get_response(URI.parse(url)) 3 sequence = JSON.parse(resp.body)

Ruby Client

@rustyio

Protocol Buffers - Client

23

Ruby Client 1 require 'beefcake' 2 3 class SequenceRequest 4 include Beefcake::Message 5 required :n, :int32, 1 6 end 7 8 class SequenceResponse 9 include Beefcake::Message 10 repeated :sequence, :string, 1 11 end 12 13 req = SequenceRequest.new(:n => 5) 14 Socket.tcp("localhost", 8003) do |socket| 15 socket.write(req.encode) 16 m = socket.read 17 return SequenceResponse.decode(m) 18 end

@rustyio

BERT-RPC - Client

24

Ruby Client 1 require 'bert-rpc' 2 3 svc = BERTRPC::Service.new('localhost', 9999) 4 sequence = svc.call.ernie_sequence.sequence(5)

@rustyio

Show me the Server Code!

25

@rustyio

REST / JSON - Server

26

1 -module(spooky_sequence). 2 -behaviour(spooky). 3 -export([init/1, get/2]). 4 5 init([])-> 6 [{port, 8002}]. 7 8 get(_Req, ["sequence", Num])-> 9 case sequence:sequence(Num) of 10 {ok, List} -> 11 {200, mochijson2:encode(List)}; 12 {error, Error} -> 13 {500, io_lib:format("~w", [Error])} 14 end; 15 get(_Req, _)-> 16 {400, "Usage: /sequence/:Num:"}.

Spooky Server

@rustyio

REST / JSON - Server

27

Webmachine Server

1 -module(webmachine_sequence). 2 -export([init/1, content_types_provided/2, to_json/2]). 3 -include_lib("webmachine/include/webmachine.hrl"). 4 5 -record(ctx, { list }). 6 7 init([]) -> 8 {ok, #ctx {}}. 9 10 content_types_provided(RD, Ctx) -> 11 Types = [{"application/json", to_json}], 12 {Types, RD, Ctx}. 13 14 to_json(RD, Ctx) -> 15 {ok, List} = sequence:sequence(N), 16 Body = mochijson2:encode(List), 17 {Body, RD, Ctx}.

@rustyio

Protocol Buffers - Server

28

Erlang Server 1 %% Erlang RPC Demo 2 %% Copyright (c) 2011 Rusty Klophaus (@rustyio) 3 %% See MIT-LICENSE for licensing information. 4 5 -module(protobuff_server). 6 -behaviour(gen_server). 7 8 -export([ 9 start_link/0, 10 set_socket/2, 11 init/1, 12 handle_call/3, 13 handle_cast/2, 14 handle_info/2, 15 terminate/2, 16 code_change/3, 17 encode/1, 18 decode/1]). 19 20 -record(state, { sock }). 21 22 -include("rpc_demo_pb.hrl"). 23 24 %% =================================================================== 25 %% Public API 26 %% =================================================================== 27 28 start_link() -> 29 gen_server:start_link(?MODULE, [], []). 30 31 set_socket(Pid, Socket) -> 32 gen_server:call(Pid, {set_socket, Socket}). 33 34 init([]) -> 35 {ok, #state{}}. 36 37 handle_call({set_socket, Socket}, _From, State) -> 38 inet:setopts(Socket, [{active, once}, {packet, 4}, {header, 1}]), 39 {reply, ok, State#state{sock = Socket}}. 40 41 handle_cast(_Msg, State) -> 42 {noreply, State}. 43 44 handle_info({tcp_closed, Socket}, State=#state{sock=Socket}) -> 45 {stop, normal, State}; 46 handle_info({tcp_error, Socket, _Reason}, State=#state{sock=Socket}) -> 47 {stop, normal, State}; 48 handle_info({tcp, _Sock, MsgData}, State=#state{sock=Socket}) -> 49 Msg = decode(MsgData), 50 case process_message(Msg, State) of 51 {pause, NewState} -> 52 ok; 53 NewState -> 54 inet:setopts(Socket, [{active, once}]) 55 end, 56 {noreply, NewState};

57 handle_info({tcp, _Sock, _Data}, State) -> 58 %% req =/= undefined: received a new request while another was in 59 %% progress -> Error 60 lager:error("Received a new PB socket request" 61 " while another was in progress"), 62 {stop, normal, State}; 63 64 handle_info(_, State) -> % Ignore any late replies from gen_servers/messages from fsms 65 {noreply, State}. 66 67 terminate(_Reason, _State) -> 68 ok. 69 70 code_change(_OldVsn, State, _Extra) -> {ok, State}. 71 72 %% =================================================================== 73 %% Handle PB Messages 74 %% =================================================================== 75 76 process_message(#sequencerequest { n = N }, State) -> 77 case sequence:sequence(N) of 78 {ok, List} -> 79 Resp = #sequenceresponse { sequence = List }, 80 send_msg(Resp, State); 81 {error, Reason} -> 82 Msg = io_lib:format("~w", [Reason]), 83 Resp = #sequenceerror { message=Msg }, 84 send_msg(Resp, State) 85 end. 86 87 %% Send a message to the client 88 send_msg(Msg, State) -> 89 Pkt = encode(Msg), 90 gen_tcp:send(State#state.sock, Pkt), 91 State. 92 93 encode(Msg) -> 94 MsgType = element(1, Msg), 95 [msg_code(Msg) | rpc_demo_pb:iolist(MsgType, Msg)]. 96 97 decode([MsgCode|MsgData]) -> 98 MsgType = msg_type(MsgCode), 99 rpc_demo_pb:decode(MsgType, MsgData). 100 101 msg_code(#sequencerequest {}) -> 1; 102 msg_code(#sequenceresponse {}) -> 2; 103 msg_code(#sequenceerror {}) -> 3; 104 msg_code(Other) -> 105 throw({unknown_pb_type, Other}). 106 107 msg_type(1) -> sequencerequest; 108 msg_type(2) -> sequenceresponse; 109 msg_type(3) -> sequenceerror; 110 msg_type(Other) -> 111 throw({unknown_pb_code, Other}).

@rustyio

BERT-RPC - Server

29

Ernie Server - Erlang BERT-RPC Server 1 %% FILE: ernie.config 2 [ 3 {module, ernie_sequence}, 4 {type, native}, 5 {codepaths, []} 6 ].

1 -module(ernie_sequence). 2 -export([sequence/1]). 3 4 sequence(N) -> 5 sequence:sequence(N).

@rustyio

BenchmarksHow fast is it?

30

@rustyio

Benchmarks

31

DisclaimersBenchmarking is hardThe shape of your data matters (size and complexity)Speed is not the only objective

@rustyio

Benchmarks:Encoding Speed

32

@rustyio

Encoding Speed - 5 items

33

BERT JSON PB

@rustyio

Encoding Speed - 50 items

34

BERT JSON PB

@rustyio

Encoding Speed - 500 items

35

BERT JSON PB

@rustyio

Encoding Speed - 5,000 items

36

BERT JSON PB

@rustyio

Encoding Speed - 50,000 items

37

BERT JSON

Sad Protocol Buffers :(

@rustyio

Encoding Speed - Ruby

38

| 5 | 50 | 500 | 5,000-----+---------+---------+---------+----------JSON | 0.03 ms | 0.77 ms | 0.47 ms | 4.21 ms PB | 0.07 ms | 0.60 ms | 7.37 ms | 197.24 msBERT | 0.08 ms | 0.52 ms | 4.97 ms | 52.42 ms

@rustyio

Encoding SpeedInterpretations

BERT wins in Erlang, because it’s native.JSON wins in Ruby.Protocol Buffers is slow all around for complex data. • Note: This is different from large data.

39

@rustyio

Benchmarks:Single Hop

40

$ ping dell.local

PING dell.local (192.168.2.2): 56 data bytes64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.490 ms

@rustyio

Benchmarks

41

More DisclaimersFirst approach exhausted TCP connections“Fixed” by tunneling connections, which impacts results

@rustyio

Operation Performance - 5 items

42

BERT SPOOKYPB PB WEBMACHINE

⬆ ⬆

@rustyio

Operation Performance - 50 items

43

BERT SPOOKYPB PB WEBMACHINE

⬆ ⬆

@rustyio

Operation Performance - 500 items

44

BERT SPOOKYPB PB WEBMACHINE

⬆ ⬆⬆

@rustyio

Operation Performance - 5,000 items

45

BERT SPOOKYPB PB WEBMACHINE

⬆

@rustyio

Encoding SpeedInterpretations

Performance is fairly even for simple data.PB is clear loser for data with lots of items.Webmachine pays a tax for considering entire HTTP decision tree.

Take these with a grain of salt, everything is tunneled over SSH.

46

@rustyio

Webmachine - HTTP Decision Tree

47

@rustyio

Benchmarks:Multiple Hops

48

$ ping rusty.io PING rusty.io (173.203.217.46): 56 data bytes64 bytes from 173.203.217.46: icmp_seq=0 ttl=53 time=49.550 ms

@rustyio

Operation Performance - 5 items

49

BERT SPOOKYPB PB WEBMACHINE

@rustyio

Operation Performance - 50 items

50

BERT SPOOKYPB PB WEBMACHINE

@rustyio

Operation Performance - 500 items

51

BERT SPOOKYPB PB WEBMACHINE

@rustyio

Operation Performance - 5,000 items

52

BERT SPOOKYPB PB WEBMACHINE

@rustyio

Operation Performance - Multiple HopsInterpretations

No clear winner.Network speed / variability is the bottleneck.

Take these with a grain of salt, everything is tunneled over SSH.

53

@rustyio

The ResultsRecommendations

Start with REST / JSONOptimize for performance with Protocol BuffersGet adventurous with BERT-RPC• Easy to set up == Easy to back out

Get Involved!Protocol Buffers NIF?JSON NIF?General Ernie (BERT-RPC) improvements:• Re-use connections, better packaging & documentation

54

@rustyio

Honorable MentionsOther RPC Frameworks:

Thrift - http://thrift.apache.org/ (Originally Facebook)

Avro - http://avro.apache.org/docs/current/

XML-RPC

CORBA - http://www.erlang.org/doc/man/corba.html

UBF - http://www.sics.se/~joe/ubf/site/home.html

MsgPack - http://msgpack.org/

Etch - http://incubator.apache.org/projects/etch.html

ASN.1 - http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One

Erlang <-> Other Code:Erlang "ports" - integration via stdin/stdout.

Bifs - Extend Erlang with new functions.

"Fake" Erlang Nodes:JInterface (it is... not great)

C Nodes - http://www.erlang.org/doc/tutorial/cnode.html

55

@rustyio

Related Talks• Anton Lavrik - Piqi-RPC: exposing Erlang services via JSON, XML and Google

Protocol Buffers over HTTPhttp://www.erlang-factory.com/conference/SFBay2011/speakers/AntonLavrik

• Tom Preston-Werner - BERT is to Erlang as JSON is to JavaScript http://www.erlang-factory.com/conference/ErlangUserConference2009/speakers/TomPrestonWerner

• Todd Lipcon - Thrift Avro/Erlang Bindingshttp://www.erlang-factory.com/conference/SFBay2010/speakers/toddlipcon

• Cliff Moon - Building Polyglot Distributed Systems with Jinterface http://www.erlang-factory.com/conference/SFBay2011/speakers/CliffMoon

• Kresten Krab Thorup - Erjang - A JVM-based Erlang VMhttp://www.erlang-factory.com/conference/SFBay2010/speakers/KrestenKrabThorup

• Yurii Rashkovskii - Beam.JS: Erlang meets JavaScripthttp://www.erlang-factory.com/conference/SFBay2011/speakers/YuriiRashkovskii

56

@rustyio

Thanks!Questions?

57