nps: a non-interfering web prefetching system ravi kokku, praveen yalagandula, arun venkataramani,...

NPS: A Non-interfering Web Prefetching System

Ravi Kokku, Praveen Yalagandula,

Arun Venkataramani, Mike Dahlin

Laboratory for Advanced Systems Research

Department of Computer Sciences

University of Texas at Austin

April 19, 2023 Department of Computer Sciences, UT Austin 2

Summary of the Talk

Prefetching should be done aggressively, but safely

Safe: Non-interference with demand requests

Contributions:A self-tuning architecture for web prefetching

• Aggressive when abundant spare resources• Safe when scarce resources

NPS: A prototype prefetching system• Immediately deployable


Outline

Prefetch aggressively as well as safely

Motivation Challenges/principles NPS system design Conclusion


What is Web Prefetching?

Speculatively fetch data that will be accessed in the future

Typical prefetch mechanism [PM96, MC98, CZ01]

Demand Requests

Responses + Hint Lists

Prefetch Requests

Client Server

Prefetch Responses


Why Web Prefetching?

Benefits [GA93, GS95, PM96, KLM97, CB98, D99, FCL99, KD99, VYKSD01, …]

Reduces response times seen by users Improves service availability

Encouraging trendsNumerous web applications getting deployed

• News, banking, shopping, e-mail…

Technology is improving rapidly capacities and prices of disks and networks

Prefetch Aggressively


Why doesn’t everyone prefetch?

Extra resources on servers, network and clients Interference with demand requests

Two types of interference• Self-Interference– Applications hurt themselves• Cross-Interference– Applications hurt others

Interference at various components• Servers – Demand requests queued behind prefetch• Networks – Demand packets queued or dropped• Clients – Caches polluted by displacing more useful data


Example: Server Interference

Common load vs. response curveConstant-rate prefetching reduces server capacity

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

100 200 300 400 500 600 700 800

Demand Connection Rate (conns/sec)

DemandPfrate=1

Pfrate=5

Avg

. Dem

and

Res

pons

e T

ime

(s)

Prefetch Aggressively, BUT SAFELY


Outline

Prefetch aggressively as well as safely

Motivation Challenges/principles

Self-tuningDecoupling prediction from resource managementEnd-to-end resource management

NPS system design Conclusion


Goal 1: Self-tuning System

Proposed solutions use “magic numbers”Prefetch thresholds [D99, PM96, VYKSD01, …]Rate limiting [MC98, CB98]

Limitations of manual tuningDifficult to determine “good” thresholds

• Good thresholds depend on spare resources

“Good” threshold varies over timeSharp performance penalty when mistuned

Principle 1: Self-tuningPrefetch according to spare resources Benefit: Simplifies application design


Goal 2: Separation of Concerns

Prefetching has two components Prediction – What all objects are beneficial to prefetch? Resource management –How many can we actually prefetch?

Traditional techniques do not differentiate Prefetch if prob(access) > 25% Prefetch only top 10 important URLs Wrong Way! We lose the flexibility to adapt

Principle 2: Decouple prediction from resource management Prediction: Application identifies all useful objects

• In the decreasing order of importance

Resource management: Uses Principle 1• Aggressive – when abundant resources

• Safe – when no resources


Goal 3: Deployability

Ideal resource management vs. deployability Servers

• Ideal: OS scheduling of CPU, Memory, Disk…• Problem: Complexity – N-Tier systems, Databases, …

Networks• Ideal: Use differentiated services/ router prioritization• Problems: Every router should support it

Clients• Ideal: OS scheduling, transparent informed prefetching• Problem: Millions of deployed browsers

Principle 3: End-to-end resource management Server – External monitoring and control Network – TCP-Nice Client – Javascript tricks


Outline

Prefetch Aggressively as well as safely

Motivation Principles for a prefetching system

Self-tuning Decoupling prediction from resource management End-to-end resource management

NPS prototype design Prefetching mechanism External monitoring TCP-Nice Evaluation

Conclusion


Prefetch Mechanism

Client

PrefetchServer

DemandServer

Munger

Demand Requests

Hint Lists

Prefetch Requests

Fileset

HintServer

Server m/c

1. Munger adds Javascript to html pages

2. Client fetches html page3. Javascript on html page fetches hint list

4. Javascript on html page prefetches objects


End-to-end Monitoring and Control

Principle: Low response times server not loaded Periodic probing for response times Estimation of spare resources (budget) at server – AIMD Distribution of budget

• Control the number. of clients allowed to prefetch

MonitorHintServer

DemandServer

ClientGET http://repObj.html

200 OK…

while(1) { getHint( ); prefetchHint( ); }

if (budgetLeft) send(hints);else send(“return later”);


Monitor Evaluation (1)

End-to-end monitoring makes prefetching safe

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 100 200 300 400 500 600 700 800

Avg

De

ma

nd

Re

spo

nse

Tim

e(se

c)


Monitor

Manual tuning,Pfrate=1

Manual tuning, Pfrate=5

No-Prefetching



Manual tuning is too damaging at high load

0

20

40

60

80

0 100 200 300 400 500 600 700 800

Ba

ndw

idth

(M

bp

s)


pfrate=1Prefetch:

Demand: pfrate=1

No-Prefetching



Manual tuning too timid or too damagingEnd-to-end monitoring is both aggressive and safe

0

20

40

60

80

0 100 200 300 400 500 600 700 800

Ba

ndw

idth

(M

bp

s)


pfrate=1Prefetch:

Demand: pfrate=1

Demand:Monitor

No-Prefetching

Prefetch:Monitor


Network Resource Management

Demand and prefetch on separate connections Why is this required?

HTTP/1.1 persistent connections In-order delivery of TCPSo prefetch affects demand

How to ensure separation?Prefetching on a separate server port

How to use the prefetched objects?Javascript tricks – In the paper


Network Resource Management

Prefetch connections use TCP Nice TCP Nice

A mechanism for background transfersEnd-to-end TCP congestion controlMonitors RTTs and backs-off when congestionPrevious study [OSDI 2002]

• Provably bounds self- and cross-interference• Utilizes significant spare network capacity

Server-side deployable


Measure avg. response times for demand reqs.Compare with No-Prefetching and Hand-tuned

Experimental setup

End-to-end Evaluation

PrefSvrApache:8085

DemandSvrApache: 80

Fileset

HintSvrPPM predict

Clienthttperf

NetworkCable modem,

Abilene

TraceIBM server


Prefetching with Abundant Resources

Both Hand-tuned and NPS give benefitsNote: Hand-tuned is tuned to the best

24.3

37.3

11.8

19.114.5

18

0

10

20

30

40

50

Cable modem Abilene

Av

g. D

em

. Re

sp. T

ime

(m

s)

No prefetching Hand-tuned NPS


Tuning the No-Avoidance Case

Hand-tuning takes effort NPS is self-tuning

24.3 23.3

16.4

11.815.7

18.3 19.1

14.5

0

10

20

30

Cable modem

Av

g. D

em

. Re

sp. T

ime

(m

s)

No prefetching T=0.5 T=0.2 T=0.1

T=0.05 T=0.02 T=0.01 NPS


Prefetching with Scarce Resources

Hand-tuned damages by 2-8x NPS causes little damage to demand

41.6 31.6

94.6

221.1

31.8 41.1

0

50

100

150

200

250

Loaded Server Loaded Network

Av

g. D

em

. Re

sp. T

ime

(m

s)



Conclusions

Prefetch aggressively, but safely Contributions

A prefetching architecture• Self-tuning• Decouples prediction from resource management• Deployable – few modifications to existing infrastructure

Benefits• Substantial improvements with abundant resources• No damage with scarce resources

NPS prototype http://www.cs.utexas.edu/~rkoku/RESEARCH/NPS/


Thanks


Prefetching with Abundant Resources

Both Hand-tuned and NPS give benefitsNote: Hand-tuned is tuned to the best

24.337.3

134.9

11.8 19.1

73.3

14.5 18

75.5

020406080

100120140160

Cable modem Abilene Trans-atlantic

Av

g. D

em

. Re

sp. T

ime

(m

s)



Client Resource Management

Resources – CPU, memory and disk caches Heuristics to control cache pollution

Limit the space prefetch objects takeShort expiration time for prefetched objects

Mechanism to avoid CPU interferenceStart prefetching after all demand done

• Handles self-interference – more common case

What about cross-interference?• Client modifications might be necessary

nps: a non-interfering web prefetching system ravi kokku, praveen yalagandula, arun venkataramani,...

Documents

demand requestscontributions

web prefetchingaggressive

resource managementend

resource managementprediction

demand response time

abundant spare resourcessafe

selftuning architecture

spare resources benefit