The Ruby GC is your friend!

Let's get together and greet him+

Case study on studyabroad101.com

@sdwolf

About StudyAbroad101

http://www.studyabroad101.com/

Reviews for study abroad programs.Rails 3.2 and Ruby 1.9.3 (now 2.1.6) Hosted on Heroku.

In the beginning

*not actual image (it was lost in the passage of time)

Yes, the brown area is garbage collection time... but why???

Common (anti)pattern:

These were all over the place! (also no server side pagination and a lot of N+1's).

A quick fix?

Tweak the Ruby GC ENV Vars.

Ruby 1.9.3 and 2.0.0 GC

Mark and Sweep with 1 generation. (2.0.0 is Copy on Write Friendly)

3 ENV Vars:● RUBY_GC_MALLOC_LIMIT● RUBY_HEAP_MIN_SLOTS● RUBY_FREE_MIN

RUBY_GC_MALLOC_LIMIT

Default: 8_000_000 (~ 8 MB).

What it does: Triggers a GC run after X bytes were allocated.

How to tweak: The more memory you allocate, the higher this value should be. To low and you will trigger to many sweeps, to high and you will have memory and sweep spikes.

Ref:● https://github.com/ruby/ruby/blob/v1_9_3_551/gc.c#L762● https://github.com/ruby/ruby/blob/v2_0_0_645/gc.c#L3487

RUBY_HEAP_MIN_SLOTS

Default: 10_000 What it does: Defines the initial (and minimum) number of heap pages and slots to allocate. Each page occupies 16384 bytes (16 KB). Each slot occupies 40 bytes. That means there are 409 slots per page. 10_000 / 409 = 24 pages allocated = 9816 slots allocated. Note that the default allocation happens even if you change the value.

How to tweak: Set big enough to not need to increase the heap size during runtime, and in multiples of 409... if you want to (OCD nightmare).

Ref:● https://github.com/ruby/ruby/blob/v1_9_3_551/gc.c#L1146-L1150● https://github.com/ruby/ruby/blob/v2_0_0_645/gc.c#L599-L603

RUBY_FREE_MIN

Default: 4096

What it does: Allocates more heap (hard-coded at 80% of used heap) if there are less than X free slots after a sweep.

How to tweak: Make sure you never need it, unless you can afford a 80% increase in memory. If you really must use it, balance it with RUBY_HEAP_MIN_SLOTS so it gets triggered only in exceptional cases (heavy duty requests or memory leaks) and have a worker killer ready to clean it up. Remember, the heap does not shrink on it's own!

Ref:● https://github.com/ruby/ruby/blob/v1_9_3_551/gc.c#L2227● https://github.com/ruby/ruby/blob/v2_0_0_645/gc.c#L1993

The change:

RUBY_GC_MALLOC_LIMIT:RUBY_HEAP_MIN_SLOTS:RUBY_FREE_MIN:

BEFORE AFTER

8_000_00010_000

4090

60_000_000600_000

60_000

The result:

Before:

After:

The other result:

I was mind-blown!

And so the time passed...

And the code was cleaned, ruby upgraded to 2.1.6, then came the release!

GC time is so low even without any modifications, what else is there to do?Well... memory management!

Ruby 2.1 and 2.2 GC

Mark and Sweep with 2 generations. Also the heap SHRINKS (YAY!).

11 ENV Vars:● RUBY_GC_HEAP_FREE_SLOTS (replaces RUBY_FREE_MIN)● RUBY_GC_HEAP_INIT_SLOTS (replaces RUBY_HEAP_MIN_SLOTS)● RUBY_GC_HEAP_GROWTH_FACTOR● RUBY_GC_HEAP_GROWTH_MAX_SLOTS● RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR● RUBY_GC_MALLOC_LIMIT● RUBY_GC_MALLOC_LIMIT_MAX● RUBY_GC_MALLOC_LIMIT_GROWTH_FACTOR● RUBY_GC_OLDMALLOC_LIMIT● RUBY_GC_OLDMALLOC_LIMIT_MAX● RUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR

RUBY_GC_HEAP_INIT_SLOTS

Replaces: RUBY_HEAP_MIN_SLOTS

Default: 10_000

What it does: Same as before.

How to tweak: Same as before.

Ref:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L5747● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L7014

RUBY_GC_HEAP_GROWTH_MAX_SLOTS

Default: 0 (disabled).

What it does: Never allocate more than X slots at once (if X greater than 0).

How to tweak: Measure and set to a positive value if your application allocates to much heap at once.

Ref:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L1166● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L1513

RUBY_GC_HEAP_FREE_SLOTSReplaces: RUBY_FREE_MIN

Default: 4096

What it does: The minimum amount of free slots “Y” is set to the greater value between “X” (the ENV Var value) and 30% of the total number of slots but not more than 80% of the total number of slots. If the GC sweep did not clear at least Y slots we have 2 cases:- minor GC and haven't done a major GC in 2 (3 for v2.2) sweeps then we schedule a major GC sweep.- otherwise increase the heap size with the difference (in pages, at least 1) trimmed by RUBY_GC_HEAP_GROWTH_MAX_SLOTS.The 30% and 80% values are hard-coded in v2.1 and constants in v2.2. They may become ENV Vars one day.

How to tweak: Leave it alone unless you need it greater than 30%.

Ref:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L2959● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L5074

RUBY_GC_HEAP_GROWTH_FACTOR

Default: 1.8 What it does: Allocate X times the amount of used slots when increasing the heap size, trimmed by RUBY_GC_HEAP_GROWTH_MAX_SLOTS.

How to tweak: This works together with RUBY_GC_HEAP_GROWTH_MAX_SLOTS.If you want to increase the heap by a percentage, set it to the needed value (usually lower than 1.8) and set RUBY_GC_HEAP_GROWTH_MAX_SLOTS to a big number (or 0).If you want to increase the heap by a fixed number of slots, set this to a high percentage and let RUBY_GC_HEAP_GROWTH_MAX_SLOTS handle the rest. Ref:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L1165● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L1509

RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR

Default: 2.0 What it does: Checked when finished marking, if the old generation grew by this multiplier or higher since last full sweep then a full sweep is scheduled.

How to tweak: Lower means more full sweeps, higher means more memory consumption.

Ref:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L4996● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L5096

What happens with objects bigger than 40 bytes?

They are allocated more memory INSIDE the heap but OUTSIDE of the pages.

How to deal with that allocated memory?

For the new generation, 3 ENV Vars:● RUBY_GC_MALLOC_LIMIT

Default: 16 * 1024 * 1024 (16MB)What it does: Minimum malloc limit to trigger a minor GC.

● RUBY_GC_MALLOC_LIMIT_MAXDefault: 32 * 1024 * 1024 (32MB) (0 means disabled)What it does: Maximum value for the malloc limit.

● RUBY_GC_MALLOC_LIMIT_GROWTH_FACTORDefault: 1.4What it does: Sets the rate at which the malloc limit grows.

The limit is initially set at the minimum value, triggers a minor GC if the limit is exceeded when allocating memory. Before a sweep, if the malloced memory X grew more than the limit, the new limit is set to X times the factor, trimmed by the min/max. Else the limit is shrunk by 2%.

How to tweak: Increase or decrease the maximum value as needed, but leave the rest as is.

Refs:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L2888-L2900● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L5798-L5810

For the old generation, 3 ENV Vars:● RUBY_GC_OLDMALLOC_LIMIT

Default: 16 * 1024 * 1024 (16MB)What it does: Minimum malloc limit to trigger a major GC.

● RUBY_GC_OLDMALLOC_LIMIT_MAXDefault: 128 * 1024 * 1024 (128MB) (can't be disabled)What it does: Maximum value for the malloc limit.

● RUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTORDefault: 1.2What it does: Sets the rate at which the malloc limit grows.

During a minor GC if the limit is exceeded, a major GC is scheduled and the limit is increased times the growth factor. During a major GC, the limit is reduced by the growth factor if the GC run was not triggered by exceeding the OLD malloc limit.

How to tweak: Increase or decrease the maximum value as needed, but leave the rest as is.

Refs:● https://github.com/ruby/ruby/blob/v2_1_6/gc.c#L2916-L2945● https://github.com/ruby/ruby/blob/v2_2_2/gc.c#L5826-L5855

Final setup:

RUBY_GC_HEAP_FREE_SLOTSRUBY_GC_HEAP_INIT_SLOTSRUBY_GC_HEAP_GROWTH_FACTORRUBY_GC_HEAP_GROWTH_MAX_SLOTSRUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTORRUBY_GC_MALLOC_LIMITRUBY_GC_MALLOC_LIMIT_MAXRUBY_GC_MALLOC_LIMIT_GROWTH_FACTORRUBY_GC_OLDMALLOC_LIMITRUBY_GC_OLDMALLOC_LIMIT_MAXRUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR

BEFORE AFTER

409610_000

1.80

2.016_777_21633_554_432

1.416_777_216

134_217_7281.2

100_000860_000

1.2220_000

2.020_000_00090_000_000

1.420_000_000

140_000_0001.2

Sorry for the long presentation...

Here are some Ruby Potatoes: