altitude sf 2017: optimizing your hit rate

Optimizing Your Cache Hit Ratio

Léon BrocardSales Engineer | Fastly

Paolo AlvaradoCustomer Support Manager | Fastly

Léon Brocard

Systems Engineer

Fastly London

Paolo Alvarado

Customer Support Manager

Fastly Tokyo

Cache hit ratioWhy do we cache?Caching on FastlyHTTP cachingOverriding cachingCaching patternsFuture

This is a talk workshop

Blue slidemeans

interactive

Cachehit

ratio

Where is“Cache hit ratio” on

https://www.fastly.com/

https://manage.fastly.com/

[email protected]

Name & Domainhitrate2017<initials>.global.ssl.fastly.net

Address35.197.9.99

No TLS, Port3000

Name & Domainhitrate2017lb.global.ssl.fastly.net

Address35.197.9.99

No TLS, Port3000

http://hitrate2017lb.global.ssl.fastly.net/

Why do we cache?

Decrease pageload time

Level Capacity Latency

Can on my desk

1 can ~1s

My fridge 10 cans ~30s

Local shop 500 cans ~5m

Supermarket 10k cans ~90m


Cash in my wallet

$50 ~1s

Local ATM $200 ~5m

Bank $1,000 ~20m


Plate 1 serving ~1s

Serving dish 10 servings ~1m

Kitchen 100 servings ~5m


L1 cache 64 KiB per core ~1ns

L2 cache 256 KiB per core ~4ns

L3 cache Up to 2 MiB per core, shared

~20ns

Main memory Up to 64 GB ~100ns


Browser ~Megabytes ~0.1ms

CDN ~Gigabytes ~5ms

Origin ~Terabytes ~500ms

Cache?

L'écureuil roux cache ses provisions pour l'hiver

Hit Miss

Scenario Origin requests / sec

EC2 instances

No CDN 2,000 40

50% CHR 1,000 20

80% CHR 400 8

90% CHR 200 4

95% CHR 100 2

Representativetime periodtraffic

Why is latency so important?

Wi-Fi Mobile

HTTPTCP/IP

GET /technology/2016/jul/15/how-the-internet-was-invented-1976-arpa-kahn-cerf HTTP/1.1Host: www.theguardian.com

HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8Content-Length: 110502

<!DOCTYPE html>...

Visithitrate2017lb.global.ssl.fastly.net

View ▶ Developer ▶ Developer Tools

Network tab

Reload, ⌘R, shift-⌘R

1.5s 2.0s 3.5s 21.0s

www.webpagetest.orgURL: docs.fastly.comEC2 location with low pending testsConnection DSL, capture video, keep test private

Here’s one I prepared earlier:

http://fastly.us/2sybe4q

Back to developer toolsRight click on table headingResponse headers ▶ Add “Cache-Control”Response headers ▶ Manage header columns ▶ Add custom header → “X-Cache”

if users double their bandwidth without reducing their RTT significantly, the effect on Web Browsing will be a minimal improvement. However, decreasing RTT, regardless of current bandwidth always helps make web browsing faster

How can we decrease RTT?

How can we decrease RTT?

latency?

Caching

Visithttps://www.fastly-debug.com

/

HTTPcaching

1996 RFC 1945 HTTP/1.0

1997 RFC 2068 HTTP/1.1

1999 RFC 2616 HTTP/1.1

2014 RFC 7230 HTTP/1.1: Message Syntax and Routing

2014 RFC 7231 HTTP/1.1: Semantics and Content

2014 RFC 7232 HTTP/1.1: Conditional Requests

2014 RFC 7233 HTTP/1.1: Range Requests

2014 RFC 7234 HTTP/1.1: Caching

2014 RFC 7235 HTTP/1.1: Authentication

2015 RFC 7540 HTTP/2

Fresh Expired

Last-Modified: Fri, 13 Nov 2015 14:11:55 GMT

If-Modified-Since: Fri, 13 Nov 2015 13:58:51 GMT

HTTP/1.1 304 Not Modified

ETag: “3e8d1bf998b31cb811264f4789eac005”

If-None-Match: “3e8d1bf998b31cb811264f4789eac005”

HTTP/1.1 304 Not Modified


Network tabBrowser cachemax-age=60 tabFirst time. Second time.

Cache-Control: max-age=60Date: Thu, 25 May 2017 10:16:09 GMTExpires: Thu, 25 May 2017 10:16:58 GMT

Cache-Control: max-age=1800

Cache-Control: s-max-age=1800, max-age=60Surrogate-Control: max-age=1800


Network tabBrowser cache⌘RCache-Control:(none) / private / max-age=60

Tool time

Install Chrome Extension “Browser Cache Hit Ratio”...

https://fastly.us/browser-cachehitratio

Install Chrome Extension “Fastly Cache Hit Ratio”...

https://fastly.us/cachehitratio

Install Chrome Extension “Modheader”...

https://fastly.us/modheader

What’s myTTL again?

Changingcaching

On the origin with

HTTP headers

Override in Fastly

No Cache-Control header

Fastly Service ▶ Configuration ▶ Clone active ▶ Settings ▶ Fallback TTL ▶ 10 ▶ Save ▶ Activate ▶ Purge ▶ Purge All ▶ Purge

Cache-Control: private

Cache-Control: max-age=60

Varnish Configuration Language

defaultTTL?

if ( beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(s-maxage|max-age)" ) { # keep the ttl here } else { # apply the default ttl set beresp.ttl = 10s;}

if (req.url == "/dogs/max-age=60") { set beresp.http.Cache-Control = "max-age=10";}

if (req.url == "/dogs/max-age=60") { set beresp.ttl = 10s; set beresp.http.Cache-Control = "max-age=10";}

sub vcl_fetch { if (beresp.http.Cache-Control) { if (req.url.path ~ "^/users/flair/") { set beresp.http.Cache-Control = "public, max-age=180"; } else { set beresp.http.Cache-Control = "private"; }}

Here’s an example of where VCL comes in very handy. Recently we deployed .NET 4.6.2 which had a very nasty bug that set max-age on cache responses to over 2000 years. The quickest way to mitigate this for all of our services affected was to override that cache header as-needed at the edge. As I write this, the following VCL is active:

What’s in a cache key?

req.http.hostThe incoming request’s host header

req.urlThe full URL including query string

#####GENERATION#####A generation number used for purging all

What do these URLs have in common?

● /search?query=dogs&page=1● /search?page=1&query=dogs

/search?query=dogs&page=1Fastly-Debug-Digest: 9ff1c58f06cac2fd3a48f7c105da6e1d035c770c635371f2ddabdb987818e36c

/search?page=1&query=dogsFastly-Debug-Digest:9ff1c58f06cac2fd3a48f7c105da6e1d035c770c635371f2ddabdb987818e36c

The uncacheable

“Dynamic content is really interesting”—Hooman 2014

Classically, dynamic content was uncacheable because it required interaction with a backend.

How can we cache dynamic

content?

Set-Cookie:Cache-Control: private

How do we force an update for cached

content?

Purging

Single URL PurgeInstant Purge an individual URL.

Surrogate-Key PurgeInstant Purge a particular service of items tagged with a Surrogate-Key.

Purge AllInstant Purge everything from a service.

Offloading from origins

HTTPS everywhere

http://hitrate2017lb.global.ssl.fastly.net/301 Moved Permanently redirect to:https://hitrate2017lb.global.ssl.fastly.net/

HIT-SYNTH

Strict-Transport-Security: max-age=31536000

HSTS Preload List

Serving Stale

What does a negative number here mean?

“Worst-case scenario, we can always ghost-serve. Which actually happened: our Rolling Stone legacy origin was down for three days; the only reason readers could still visit our site without noticing the problem was because we used Fastly’s stale-if-error.”

— Chris Boylan, Director of Engineering,Wenner Media

How can we purge while still

offloading?

Soft PurgeSingle URL PurgeSurrogate-Key Purge

Soft Purging sets an object’s TTL to 0s, forcing revalidation. For best results, use in combination with stale-while-revalidate and/or stale-if-error

Levels of caches

50%CHR

50%CHR

× = 75%CHR

50 50

25 25

90%CHR

90%CHR

× = 99%CHR

90 10

9 1

When the royal baby George Alexander Louis was born in 2013, the Guardian's traffic spiked from an average of 400 requests per second to over 1,000 requests per second. Fastly shielded the Guardian's origin servers from the hundreds of thousands of new requests pouring in, maintaining site uptime and delivering consistent performance for readers around the world. The company’s software architect, Matthew O’Brien, said that it’s difficult to predict news events, but Fastly helps them prepare accordingly.

Cache in browser and Fastly for different

amount of time?

Cache-Control: privateSurrogate-Control: max-age=60, stale-while-revalidate=5, stale-if-error=86400

Cache-Control: max-age=5Surrogate-Control: max-age=3600

“Fastly effectively handled the higher levels of traffic we experienced during the Apple Watch announcement.” Using Fastly’s stale-while-revalidate feature to serve readers slightly outdated content while the cached content was refreshed, WIRED knew readers would never see errors. Zack described stale-while-revalidate as one of Fastly’s “most exciting features.” He said, “It allowed us to successfully serve the live blog while constantly updating its content, ensuring our readers had access to the freshest content without seeing errors."

— Zack Tollman, Lead Engineer at WIRED

Observing cache hit ratio

X-Cache: HITX-Cache-Hits: 649406

X-Cache: HIT, HITX-Cache-Hits: 649406, 922

fastly_info.state

Jun 08 14:27:24 cache-ord1722 papertrail: MISS-CLUSTER "-" [08/Jun/2017:14:27:23 +0000] GET /recipes/?show=Beef%20brochettes HTTP/1.1 200 26

Logging atthe edge

Chris JackelSystems Engineer | Fastly

vcl_hashvs

Vary

Hash Vary

Accept-Encoding: gzip, deflate, br

Content-Encoding: gzipVary: Accept-Encoding

Future

Cache-Control: max-age=31536000, immutable

HTTP/2 enables a more efficient use of network resources and a reduced perception of latency by introducing header field compression and allowing multiple concurrent exchanges on the same connection

Service WorkersThis specification describes a method that enables applications to take advantage of persistent background processing, including hooks to enable bootstrapping of web applications while offline

The power of the network

Andrew BettsPrincipal Developer Advocate | Fastly

Emerging and future web trends

Alex RussellSoftware Engineer | Google

Summary

Cache hit ratioWhy do we cache?Caching on FastlyHTTP cachingOverriding cachingCaching patternsFuture

Run live experiments

Thanks for participatingLéon & Paolo

altitude sf 2017: optimizing your hit rate

Technology