End-to-End Arguments in System Design

J.H. Saltzer, D.P. Reed and D.D Clark

M.I.T. Laboratory for Computer Science

Presented byJimmy Pierce

Purpose of the Paper

Provide a method for which programmers can decide on the placement of functions

Provides an argument for moving functions away from the lower levels and into the application layer

This argument has become more important and apparent as systems become interconnected via communications networks

What is the end-to-end argument

Argument against placement of functions at low levels.

Why?

Only the application knows what it wants

Thus only complete and correct functionality can be performed at the application level

Therefore, providing complete functionality cannot be done at a lower level

So why implement low level functionality

Performance

Incomplete or partial functionality is often helpful for larger functionality desired by the application

Performing computation at a lower level may provide faster results than if computed at the end of a transaction

Example: Transferring large files across an error prone network

Application level reliability entails sending a checksum with data file

If checksum does not match, file is resent

Overhead incurred from retransmission exponentially grows with file size

Solution: Perform data validation at various stages

Error caught result in retransmission between segments, not whole transmission path

Overhead remains relatively constant

Application Level Functions are Still Required

Functionality at lower level does not guarantee correct functionality at application Level

Lower level functions can only perform what is within the lower level’s scope

Eg, a network driver cannot ensure that the disk properly wrote the packet data to a file

Example: MIT network system

Files are transferred across the network between computers through series of gateways

Gateways performed packet checksumming to ensure packet sent and received reliably

However, an error was present in intermediate storage mechanism on an gateway

Thus files transferred on network were corrupted

How could this problem have been avoided?

Verifying at the application level that the files were transferred correctly

A simple checksum from the source could have verified the file’s integrity

So why not provide low level functionality along withapplication functionality?

Overhead

Providing functionality incurs some overhead, not matter how slight

Low level functionality is directed towards a single use of the low level facility

Not all applications will use the low level facility in the same manner

Thus overhead will be incurred even when applications do not want to take advantage of the lower level offerings

Back to the generalization of program interfaces

Example: Interactive digital conversations, such as a cell phone

Real-time transmission and reception of data is very important

Packets can be dropped or corrupted to ensure that transmission delay is low

Human conversation are redundant in nature

The opposite is true

No need for real-time transmission of data as it can be reconstructed on the voicemail server

Data must arrive intact, user cannot ask sender to repeat their message

During a live conversation

During voicemail recording

Another Example

HP AutoRAID System

Works well on the Fast File System

Provides data reliability while improving application performance

But what if application is running on top of a logging file system?

If we do not use AutoRAID, how else can we provide hardware redundancy

Yet Another Example

The Exo-Kernel

Relies on the application and the library OS of its choice

Provides the absolute minimum

Results in very little overhead for applications

Performance enhancements reside in higher level library OS rather than in lower level kernel

Conclusions

The end-to-end argument provides a method to determine whether a feature belongs in a low level function or higher up by the application

If in doubt, implement the function at the application level