Scalable Kernel Performance for Internet Servers under Realistic Loads.

Gaurav Banga, etc...

Western Research Lab : Research Report 1998/06(Proceedings of the 1998 USENIX Annual Technical Conference)

Computer Architecture Lab. CS Dept. KAIST

2000/11/

Kim, Sung-Wan

2/16

Contents

• Introduction

• Problems of select() & ufalloc() in event-driven servers

• Scalable select() & ufalloc()

• Experimental evaluation

• Performance of a live system

• Conclusions

3/16

Introduction

• Event-driven servers– A single thread

• manage all connections– Lower context-switching & synchronization overhead

• faster than a thread-per-connection or pre-forked system– But, perform poorly under real conditions

• select() & ufalloc()

• select()– Asynchronous I/O

• ufalloc()– Allocation of a new file descriptor for a process

4/16

Problems in select() & ufalloc()

• WAN environments– Larger round-trip time and packet losses than LAN environments

– Many open connections

• select()– select() -> do_scan() -> selscan() -> soo_select()

– select_wakeup() -> do_scan() -> selscan() -> soo_select()

– soo_select()

• check to see if the condition is true

• Linear search for all opened socket

• ufalloc()– Single bitmap (first lower descriptor number)

– Too cost

5/16

Environment

• Server– AlphaStation 500(400Mhz), 192 MB of main memory

– Digital UNIX 4.0B

– Squid 1.1.11, NetCache 3.1.2c-OSF

• Client– AlphaStation 500(333Mhz)

– Digital UNIX 3.2C

– S-Client

• Network– 100Mbps FDDI

• Profiling– DCPI

6/16

• CPU times in unmodified kernel

7/16

Scalable select() & ufalloc()

• select()– READY, INTERESTED, HINTS set

– sowakeup()

• Records a hint in the HINTS sets of each of the threads in the referencing processes for which this socket is present in the INTERESTED set of the thread.

• ufalloc()– 2-level bitmap 1 0

1 1 1 1 0 0 1 1

Level 0 map

Level 1 map

INTERESTEDnew = SELECTING U INTERESTEDold

READYnew = C (INTERESTEDnew ^ (!INTERESTEDold U READYold U HINTS))READYto_user = SELECTING ^ READYnew

8/16

Experimental Evaluation- Scalability with respect to connection rate

* 750 infinitely slow connections

9/16

Experimental Evaluation- Scalability with respect to connection rate

10/16

Experimental Evaluation- Scalability with respect to connection count

11/16

Performance of a live system

• Server– A Web proxy system at DEC

– AlphaStation 500 (500 MHz), 512 MB of RAM

– Running the system for an entire day

– Proxy

• Squid

• NetCache

12/16

Performance of a live system- NetCache with caching disabled

13/16

Performance of a live system- NetCache with caching disabled

14/16

Performance of a live system- NetCache with caching enabled

15/16

Performance of a live system- NetCache with caching enabled

16/16

Performance of a live system- Squid with caching disabled

17/16

Performance of a live system- Squid with caching disabled

18/16

Performance of a live system- Squid with caching disabled

19/16

Conclusions

• WAN delays

• Linear scaling in the select() & ufalloc()– lead to excessive kernel CPU computation

• Scalable versions– improve the performance of Web servers and proxies

20/16

select(maxfd, &readfds, &writefds, …, …);

1008   for (i = 0; i < maxfd; i++) { 1009      /* Check each open socket for a handler. */ 1010      if (fd_table[i].read_handler) { 1011        if (fd_table[i].stall_until <= squid_curtime) { 1012          nfds++; 1013          FD_SET( i, &readfds); 1014        } 1015      } 1016      if (fd_table[i].write_handler) { 1017         nfds++; 1018         FD_SET(i, &writefds); 1019      } 1020   }