hot-patching a web server: a case study of asap code repair · 3 1' 2' 3' kernel....
TRANSCRIPT
![Page 1: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/1.jpg)
Hot-Patching a Web Server: a Case Study of ASAP Code Repair
Mathias Payer*, Thomas R. Gross
Department of Computer Science
ETH Zurich
* now at UC Berkeley
![Page 2: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/2.jpg)
Security Dilemma
Integrity and availability threatened by vulnerabilities
Two remedies: update or sandboxing• Security updates fix known vulnerabilities but
require service restart
• Sandboxes protect from unknown exploits but stop the service when an attack is detected
![Page 3: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/3.jpg)
DynSec in 1 Minute
Key insight: both sandboxes and dynamic update mechanisms rely on some form of virtualization
Binary Translation (BT) provides virtualization• Sandbox protects integrity
• Dynamic update mechanism protects availability
![Page 4: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/4.jpg)
DynSec in 2 Minutes
Binary Translation
ApplicationDynSec
Kernel
Patches
Loader
Patch extractionand
distribution
![Page 5: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/5.jpg)
Hot-Patching a Web Server
Analyze all security patches of Apache 2.2• From Dec 1st 2005 to Feb. 18th 2013
• Total of 49 security bugs, most are simple
• Many different classes of bugs
All vulnerabilities
Sandbox protection
Software patches
![Page 6: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/6.jpg)
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation
Conclusion
![Page 7: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/7.jpg)
Code Translation
● Translates individual basic blocks● Weave patches into translated code● Protect from security exploits
Original Code Translated Code
Binary Translator
1
2
43
1'
2'
3'
Kernel
![Page 8: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/8.jpg)
Patch Classes
Simple patch• Only few instructions change, directly patched
Patches building on DSO:• New import patch: additional library function used
• New function patch: additional function
• Additional call patch: calls to existing functions
• New String patch: new static string used
Other patches• Type change, code refactoring, heavyweight changes
![Page 9: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/9.jpg)
Patch Distribution
Most Linux distributions provide dynamic update service; piggy pack on this distribution service
• Automatically generate a dynamic patch when new package is generated
• Systems download packages and install dynamic patches to running services
• System administrators update binaries during next maintenance window
![Page 10: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/10.jpg)
Implementation
DynSec builds on TRuE/libdetox [IEEE S&P’12, ACM VEE’11]
• Patching thread injected in BT layer
• Implemented in <2000 LoC
• 48 LoC changed in TRuE to add DynSec hooks
• Supports unmodified, unaware, multi-threaded x86 applications on Linux
![Page 11: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/11.jpg)
Outline
Motivation
Patching architecture & distribution
Apache case-study• Vulnerability classes
• Distribution
Evaluation
Conclusion
![Page 12: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/12.jpg)
Apache: Vulnerability Classes
DoS XSS IL EXE HBOF lPE
AIH lDoS CSRF ACI IOF
Low 23Moderate 19Important 7
![Page 13: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/13.jpg)
DynSec Coverage
Most (45/49) vulnerabilities are hot patchable• All 7 important vulnerabilities
• 18 (out of 19) moderate vulnerabilities
• 20 (out of 23) low vulnerabilities
Patch complexity• Important patches: 4 simple, 3 DSO patches
• Moderate patches: 6 simple, 12 DSO patches
• Low patches: 10 simple, 10 DSO patches
![Page 14: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/14.jpg)
DynSec: Uncovered exploits
CVE-2007-3304 (lDoS, mod): signals to arbitrary PIDs• Heavy code refactoring
CVE-2008-0005 (XSS, low): missing UTF-7 encoding• Additional types, new functions
CVE-2012-0031 (DoS, low): scoreboard parent DoS• Type change, new functions
CVE-2012-0883 (DoS, low): insecure variable in script• Not applicable to DynSec (start-up script only)
![Page 15: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/15.jpg)
DynSec: Uncovered exploits
CVE-2007-3304 (lDoS, mod): signals to arbitrary PIDs• Heavy code refactoring
CVE-2008-0005 (XSS, low): missing UTF-7 encoding• Additional types, new functions
CVE-2012-0031 (DoS, low): scoreboard parent DoS• Type change, new functions
CVE-2012-0883 (DoS, low): insecure variable in script• Not applicable to DynSec (start-up script only)
Possibility for 4 year stride without restart
![Page 16: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/16.jpg)
Sandbox Coverage
Protects from all code-based exploits• Code injection
• Control-Flow redirection (ROP/partial JOP)
• System call policies
Unpatched protection for 11 (of 49) bugs• Two important vulnerabilities (out of 7)
• 5 moderate vulnerabilities (out of 20)
• 4 low vulnerabilities (out of 21)
![Page 17: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/17.jpg)
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation• SPEC CPU 2006 performance
• Apache performance
Conclusion
![Page 18: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/18.jpg)
Evaluation
DynSec evaluated using SPEC CPU2006• CPU: Intel Core2 Quad Q6600 @ 2.64GHz, 8GB RAM
• Ubuntu 11.04, Linux 2.6.38
• Used GCC 4.5.1 with –O2
Benchmark configurations• Native
• Sandboxing (use TRuE w/ shadow stack and checks)
• DynSec (with different patches)
![Page 19: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/19.jpg)
SPEC CPU2006: Performance
0
0.5
1
1.5
2
2.5
40
0.p
erlb
ench
40
1.b
zip
2
40
3.gc
c
42
9.m
cf
44
5.g
ob
mk
45
6.h
mm
er
45
8.s
jen
g
46
2.li
bq
uan
tum
46
4.h
26
4re
f
47
1.o
mn
etp
p
47
3.as
tar
41
0.b
wav
es
41
6.g
ames
s
43
3.m
ilc
43
4.z
eusm
p
43
5.gr
om
acs
43
6.ca
ctu
sAD
M
43
7.le
slie
3d
44
4.n
amd
45
0.s
op
lex
45
3.p
ovr
ay
45
4.ca
lcu
lix
45
9.G
emsF
DTD
46
5.to
nto
47
0.lb
m
48
2.s
ph
inx3
Mea
n
Sandbox DynSec
![Page 20: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/20.jpg)
SPEC CPU2006: Performance
0
0.5
1
1.5
2
2.5
40
0.p
erlb
ench
40
1.b
zip
2
40
3.gc
c
42
9.m
cf
44
5.g
ob
mk
45
6.h
mm
er
45
8.s
jen
g
46
2.li
bq
uan
tum
46
4.h
26
4re
f
47
1.o
mn
etp
p
47
3.as
tar
41
0.b
wav
es
41
6.g
ames
s
43
3.m
ilc
43
4.z
eusm
p
43
5.gr
om
acs
43
6.ca
ctu
sAD
M
43
7.le
slie
3d
44
4.n
amd
45
0.s
op
lex
45
3.p
ovr
ay
45
4.ca
lcu
lix
45
9.G
emsF
DTD
46
5.to
nto
47
0.lb
m
48
2.s
ph
inx3
Mea
n
Sandbox DynSec
Low performance overhead (~11%)
![Page 21: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/21.jpg)
Apache: “large” files (~250kb)
0
100
200
300
400
500
600
700
800
900
1000
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: picture.png
Native TRUE DynSec DynSec-50 DynSec-100
![Page 22: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/22.jpg)
Apache: “large” files (~250kb)
0
100
200
300
400
500
600
700
800
900
1000
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: picture.png
Native TRUE DynSec DynSec-50 DynSec-100
Less than 7% slowdown
![Page 23: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/23.jpg)
Apache: small (tiny) files (~50b)
0
0.5
1
1.5
2
2.5
3
3.5
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: index.html
Native TRUE DynSec DynSec-50 DynSec-100
![Page 24: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/24.jpg)
Apache: small (tiny) files (~50b)
0
0.5
1
1.5
2
2.5
3
3.5
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: index.html
Native TRUE DynSec DynSec-50 DynSec-100
Low performance cost for large connection counts
![Page 25: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/25.jpg)
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation
Conclusion
![Page 26: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/26.jpg)
Conclusion
Virtualization enables on-the-fly code rewriting and repair for unmodified applications
• Sandbox protects integrity
• Patches provide availability
Study shows that protecting large, long-running, and modular applications like Apache is feasible
• High coverage: 45 of 49 Apache bugs patchable
• Low performance impact: 7% for Apache 2.2
![Page 27: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/27.jpg)
Patching Architecture
DynSec thread waits for incoming patches
Patch application happens in 3 steps:• Signal all application threads to stop
• Flush all code caches
• Restart application threads
Patch is applied indirectly when code is retranslated• BT checks for every instruction if a patch is available
![Page 28: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/28.jpg)
Patch Format
The focus of DynSec is on security patches• Most security patches are only few lines of code
• Type changes and code refactoring out of scope
Patches are sets of changed instructions
Each patch may specify additional shared library for more heavyweight changes
![Page 29: Hot-Patching a Web Server: a Case Study of ASAP Code Repair · 3 1' 2' 3' Kernel. Patch Classes Simple patch • Only few instructions change, directly patched Patches building on](https://reader033.vdocuments.net/reader033/viewer/2022050607/5fae86f6fcf94416c8424cb3/html5/thumbnails/29.jpg)
Patch Extraction
Build patched application with current toolchain
Extract instruction differences between patched and unpatched version of the binary (per function)
• Changed instructions are added to patch
• Check differences in static read-only data
• Manually ensure integrity of patch (no type changes, no data changes)