optimizing native code for erlang · scheduler collapse • with riak we've seen problems in...

Optimizing Native Code for ErlangSteve VinoskiBasho [email protected]@stevevinoski

1Monday, September 22, 14

mailto:[email protected]

mailto:[email protected]

INTEGRATION, ERLANG STYLE

• External: OS processes separate from the Erlang VM

• Ports

• C Nodes

• Jinterface

• TCP/UDP/SCTP networking


INTEGRATION, ERLANG STYLE

• Internal: statically or dynamically linked into the Erlang VM

• Erlang Built-in Functions (BIFs)

• Port Drivers

• Native Implemented Functions (NIFs)


INTEGRATION EXAMPLES

• rebar uses ports for external commands like git, grep, rsync

• Erlang's inet_drv port driver

• written in C

• supports TCP, UDP, SCTP for Erlang applications

• Riak's eleveldb persistence backend is a C++ NIF


NIF DETAILS


NIF DETAILS

• Start with a regular Erlang module


NIF DETAILS


• Functions can either be stubbed out to raise errors, or have default implementations


NIF DETAILS



• Corresponding NIFs live in a dynamically loaded library


NIF DETAILS




• Module typically specifies a NIF loading function via -on_load


NIF DETAILS




• Module typically specifies a NIF loading function via -on_load

• NIFs replace Erlang functions of the same name/arity at module load time


NIF EXAMPLE

• Example module: bitwise

• Provides a function exor/2 that takes a binary and a value

• exor/2 computes an exclusive or of each byte of the binary with the argument value

• Find the code here: https://github.com/vinoski/bitwise.git


https://github.com/vinoski/bitwise.git

https://github.com/vinoski/bitwise.git

NIF EXAMPLE


EXOR/2 NIF


NOW FOR SOME BIG DATA

• 2 billion bytes


LET'S TIME OUR NIF


LET'S TIME OUR NIF

• Nearly 6 seconds!

• This is bad.


ERLANG PROCESS ARCHITECTURE


CPUCore 1

. . . . . . CPUCore N



OS + kernel threadsCPU

Core 1. . . . . . CPU

Core N



Erlang VM


Core 1. . . . . . CPU

Core N



Erlang VM


Core 1. . . . . . CPU

Core N

N1

SMPScheduler Threads

(one per core)



Erlang VM

Run Queues


Core 1. . . . . . CPU

Core N

N1


(one per core)



Erlang VM

Run QueuesProcess

Process

Process

Process

Process

Process


Core 1. . . . . . CPU

Core N

N1


(one per core)



SCHEDULING A PROCESS



• A scheduler takes a process from its run queue




• It executes it until it hits 2000 reductions (function calls) or until it waits for a message, or if it hits an emulator trap





• The process then gets scheduled out and another one chosen





• The process then gets scheduled out and another one chosen

• See Jesper Louis Andersen's scheduling description:http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-scheduling.html


THREAD PROGRESS

• Scheduler threads share some data structures

• But using traditional locks or ref counts to protect them scales poorly

• Instead, schedulers report their progress frequently to other schedulers

• Schedulers use their knowledge of other schedulers' progress to know when certain operations are safe

• For more details see https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/ThreadProgress.md


BLOCKED SCHEDULERS

• Blocking a scheduler prevents thread progress, making other schedulers wait

• Blocking a scheduler also makes it unavailable to run other processes

• A NIF shouldn't occupy a scheduler for more than 1-2 ms

• NIF reductions should also be counted properly


SCHEDULER COLLAPSE

• With Riak we've seen problems in production where schedulers go to sleep and stop executing processes

• Caused by misbehaving NIFs in Riak's storage backends interfering with normal scheduler operations

• Can also be caused by misbehaving standard Erlang functions

• See Scott Fritchie's nifwait repository, md5 branch:https://github.com/slfritchie/nifwait.git


LET'S COUNT REDUCTIONS


A MISBEHAVING NIF


A MISBEHAVING NIF

• Blocked a scheduler thread for 5.86 seconds

• And only 4 reductions


WORKAROUNDS

• Break the data into chunks

• Call exor_bad/2 repeatedly, once for each chunk

• Combine the resulting chunks into a final result


CHUNKING


CHUNKING

• Problem: how to determine optimal chunk size?

• Here, we arbitrarily chose 4MB chunks


CHUNKING RESULTS


CHUNKING RESULTS

• 476 chunks processed

• Much better reduction count of 1445

• Scheduler was never blocked (probably anyway)

• But a longer execution time of 7.87 seconds


A BETTER APPROACH

• For Erlang/OTP 17.3 (released 17 Sep 2014) I added a new NIF API function: enif_schedule_nif

• Takes a name and function pointer for a NIF, and an array of arguments to pass to it

• Schedules the argument NIF for future invocation with the specified arguments

• Allows the calling NIF to yield the scheduler


EXOR2/6

• exor2/6 is an "internal NIF" not visible to Erlang

• Works through as much of the binary as it can before its timeslice runs out

• Reports reductions using enif_consume_timeslice

• When its timeslice is up, reschedules itself via enif_schedule_nif

• Adjusts chunksize for the next iteration based on progress in each iteration


...snip...


A YIELDING NIF


A YIELDING NIF

• 5.36 seconds, fastest so far

• At over 10000 reductions, much more accurate accounting

• We yielded the scheduler 5 times


ANOTHER APPROACH:DIRTY SCHEDULERS


Erlang VM

Run Queues


Core 1. . . . . . CPU

Core N

N1


(one per core)

DIRTY SCHEDULERS



Core 1. . . . . . . . . . . . . CPU

Core N

N1

DIRTY SCHEDULERS



Core 1. . . . . . . . . . . . . CPU

Core N

N1

DIRTY SCHEDULERS

. . . . . . . . . . . . .DC1 DCN

DC: Dirty CPU Scheduler41Monday, September 22, 14


Core 1. . . . . . . . . . . . . CPU

Core N

N1

DIRTY SCHEDULERS

. . . . . . . . . . . . .DC1 DCN

DC: Dirty CPU Scheduler

Shared DC Run Queue


DIRTY SCHEDULERS

N

CPUCore 1

. . . . . . . . . . . . . CPUCore N

1 . . . . . . . . . . . . .DC1 DCN

Shared DC Run Queue


DIRTY SCHEDULERS

N

CPUCore 1

. . . . . . . . . . . . . CPUCore N

1 . . . . . . . . . . . . .DC1 DCN

Shared DC Run Queue

Shared DI/ORun Queue

DI/O NDI/O 1

DI/O: Dirty I/O Scheduler

OS + kernel threads


DIRTY SCHEDULERSShared DI/ORun Queue

DI/O NDI/O 1

DI/O: Dirty I/O Scheduler

OS + kernel threads


ENABLING DIRTY SCHEDULERS

• configure --enable-dirty-schedulers

• Your Erlang shell will print something like the following system version line:

Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:8:8] \ [ds:8:8:10] [async-threads:10] [kernel-poll:false]


USING DIRTY SCHEDULERS

• Either schedule a dirty NIF via enif_schedule_nif

• Pass a flag to indicate dirty CPU or dirty I/O scheduling

• Or specify a NIF as dirty in your ErlNifFuncs array

• Both of these are new with Erlang 17.3, replacing old experimental dirty NIF API


USING DIRTY SCHEDULERS


A DIRTY EXOR/2


A DIRTY EXOR/2

• 5.95 seconds on a dirty scheduler thread

• 8 reductions and 0 yields

• But was (almost) never on a regular scheduler

• Regular schedulers were running other jobs normally


SCHEDULE IT DIRTY


SCHEDULE IT DIRTY

• No chunking or yielding needed for dirty exor/2


SCHEDULE IT DIRTY


• But dirty schedulers are finite resources


SCHEDULE IT DIRTY



• Evil dirty NIFs can completely occupy all dirty schedulers and prevent other dirty jobs from running


SCHEDULE IT DIRTY




• A dirty NIF can use enif_schedule_nif to reschedule, yielding to allow other dirty jobs to execute


SCHEDULE IT DIRTY




• A dirty NIF can use enif_schedule_nif to reschedule, yielding to allow other dirty jobs to execute

• A NIF can use enif_schedule_nif to flip itself between regular mode and dirty mode


NEXT STEPS

• Dirty drivers already in progress

• Native processes?

• see Rickard Green's original 2011 presentation on these topics: http://www.erlang-factory.com/upload/presentations/377/RickardGreen-NativeInterface.pdf


http://www.erlang-factory.com/upload/presentations/377/RickardGreen-NativeInterface.pdf




ACKNOWLEDGEMENTS

• A huge thanks to Rickard Green of the Ericsson OTP team, who has patiently guided me in this work

• Also thanks to Sverker Eriksson of the OTP team

• And thanks to Anthony Ramine for mentioning "NIF traps" one day in the #erlang IRC channel, where I got the idea for enif_schedule_nif


THANKS

http://shop.oreilly.com/product/0636920024149.do#




optimizing native code for erlang · scheduler collapse • with riak we've seen problems in...

Documents