finding and preventing run-time error handling mistakes wes weimer george necula uc berkeley...

Finding and Preventing Run-Time Error Handling Mistakes

Wes WeimerGeorge NeculaUC Berkeley

(subliminal OOPSLA advertisement)

OSQ Retreat, May 13, 2004. #3

The Context

• Reliability in programs– Think Java or C++– But not protocols or specifications

• Run-time errors– OS resource exhaustion– Network connectivity problems– Database access errors– Disk problems

• Error-handling mistakes– Forgetting to release a lock when a disk

error occurs


Glossy Summary

• It is difficult to write error-handling code• We have an analysis for finding mistakes

– Dataflow analysis and fault model

• We found over 800 mistakes in 4 MLOC• We characterize these mistakes

– What goes wrong in off-the-shelf code?

• Existing PL features are insufficient• We propose a new feature

– “compensation stacks” track your obligations

• And back it up with case studies


Error-Handling is Significant

• IBM Study: up to two-thirds of a program can be devoted to error handling [Cri87]

• Our Survey: for Java, between 3% and 46%– Transitively reachable from catch, finally– Older, bigger programs have more handling

• Mistakes in which “applications don't properly handle error conditions that occur during normal operation” are reported as one of the top ten causes of Java web app security risks


Unfortunately, it often does not work

• Most common exception handlers:1. Do Nothing2. Print stack trace, abort program

• Higher level invariants should be restored, interface requirements should be respected – Aside from “handling” the error, the code

should clean up after itself

• Why does this not happen?– It is difficult for programmers to consider

all execution paths


Error Paths #1 – Original Code

• ptII3.0.2/ptolemy/actor/gui/JNLPUtilities.java– Special Thanks: Christopher Hylands Brooks (verified)

// Copy sourceURL to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input = new BufferedInputStream(sourceURL.openStream()); BufferedOutputStream output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) { output.write(c); } input.close(); output.close();} // line 294


Error Paths #2 – Try-Finally

// Copy sourceURL to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException {

BufferedInputStream input;BufferedOutputStream output; try { input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream(

new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) {

output.write(c); }} finally { input.close(); output.close();}

}




BufferedInputStream input;BufferedOutputStream output; try {

input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream( new FileOutputStream(destinationFile));

// The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) {

output.write(c); }} finally {

input.close(); output.close();}

}

XXX


Error Paths #4 – Nested Try-Finally// Copy sourceURL to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException {

BufferedInputStream input;BufferedOutputStream output;

input = new BufferedInputStream(sourceURL.openStream());try { output = new BufferedOutputStream(

new FileOutputStream(destinationFile)); try { // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1) {

output.write(c); } finally { output.close(); }} finally {

input.close();}

}


Where are we?

Summary and Context Error Handling is Hard: Many Paths

• Finding Error-Handling Mistakes• Characterizing Mistakes• Old & New PL Features• Case Studies• Grand Finale


Defining Error-Handling Mistakes

• Want to show that programs make mistakes when run-time errors occur– What does it mean to make a mistake?

• Safety Policy:– If a socket is opened, it must be closed

before the program terminates– If a resource is acquired, it must be released

• We consider four generic resources:– Sockets, files, streams, database locks– From a survey of catch, finally, finalize

• Program should release them along all paths, even those with run-time errors


Fault Model

• “exceptions” and “run-time errors” are highly correlated in Java [CDCF’03]– Ex: “Server crashes during call” -> UnmarshalException

• Methods come with signatures listing exceptions they can throw

• Fault Model:– A called method can terminate normally or

raise any of its declared exceptions– A called method for which we have no

signature can terminate normally or raise any exception for which there is an enclosing catch block or that is declared to escape the caller


Analysis Summary

• Build Control-Flow Graph from source– Special attention to exceptional control flow– Methods can raise declared exceptions

• Symbolically execute each method– Track outstanding resources along paths– Abstract away data values– Safety Policy determines when a resource has

been acquired and when it has been released– At joins, merge paths with equal resources

• If there is a path to the end of a method that has an outstanding resource, note it


Analysis Example: Buggy Program

try { Socket s = new Socket(); s.send(“GET index.html”); s.close(); } finally { }// close should be in finally


Analysis Example

start

new socket

send

close

end


Analysis Example

start

new socket

send

close

end

{ }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }

{ }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }

{ }

{ socket }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }

{ }

{ socket } { socket }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }

{ }


{ }


Analysis Example

start

new socket

send

close

end

{ }

{ socket }

{ }


{ } { }


Analysis Example: Path with Mistake

start

new socket

send

close

end

{ }

{ socket }

{ }


{ } { }

Works for arbitrary typestate FSMs as well. Theseexamples show simpletwo-state policies.


Report Filtering

• Sources of False Positives:1. if (sock != null) sock.close();2. sock_field_we_close_later = sock;3. return sock;

• Filter Error Reports– Whenever an error path contains one of

the above forms, we remove a resource of the appropriate type

– Removes all false positives in practice– Introduces false negatives

• (3 in a random sample of ~50 eliminated reports, suggests 30 false negatives total)


Analysis Results

Program Name Lines of Code Methods withError-Handling Mistakes

Primary Forgotten Resources

hibernate2 57k 13 DB

jaxme 58k 6 File

axion 65k 15 File

hsqldb 71k 18 DB, Strm

cayenne 86k 7 File

sablecc 99k 3 Strm

jboss 107k 40 DB, Strm

mckoi-sql 118k 37 Strm, DB

portal 162k 39 DB, File

pcgen 178k 17 File

compiere 230k 322 DB

org.aspectj 319k 27 File, Strm

ptolemy2 362k 27 File, Strm

eclipse 1.6M 126 Strm, File

… 13 others … 347k 121 File, DB

Total 3.9M 818 File, DB


Where are we?

Summary and Context Error Handling is Hard: Many Paths Finding Error-Handling Mistakes

• Characterizing Mistakes• Old & New PL Features• Case Studies• Current Work: Spec Mining• Grand Finale


Characterizing Mistakes

• Common: Protect Some Areas, But Not All– staf’s STAXMonitor class: ObjectInputStream ois = null;

try { ois = new ObjectInputStream(/* ... */); // ... } catch (StreamCorruptedException ex) { if (ois != null) { ois.close(); } showErrorDialog(/* ... */); return false;}Object obj = ois.readObject(); // no tryois.close(); // no finally


Characterizing Mistakes (#2)

• Model: aX() must be followed by cX()• try-finally not nested (osage)

– try { a1(); a2(); } finally { c2(); c1(); }

• forget a resource (quartz)– try { a1(); a2(); } finally { c1(); }

• unprotected loops (ohioedge)– for (…) { a1(); work(); c1(); }

• various: flags and early releasetry { a1(); f=0; if (…) { f=1; c1(); } // close early work(); a2(); } finally { c2(); if (!f) c1(); } // or close late



• Complicated “flag-work” is common– This example adapted from Brown’s undo

int f = 0; // flag tracks progresstry { a1(); f = 1; work(); a2(); f = 2; work(); a3(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { c3(); } catch (Exception e) {} case 2: try { c2(); } catch (Exception e) {} case 1: try { c1(); } catch (Exception e) {} }}



• This approach does not scale well– Work control flow is duplicated in cleanup

codeint f = 0; // flag tracks progress try { a1(); f = 1; work(); if (…) { did_a2 = true; a2(); f = 2; } work(); a3(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { c3(); } catch (Exception e) {} case 2: if (did_a2) { try { c2(); } catch … } case 1: try { c1(); } catch (Exception e) {} }}


Destructors / Finalizers

• Great for stack-allocated objects• Error-handling contains arbitrary code:

– Example adapted from undo, which has 17 unique cleanup actions, one 34 lines long

• Called by garbage collector– Too late! Programmers often use flags

specifically to free resources early

• No ordering guarantees– Socket sock = new Socket();– Stream strm = new Stream(sock); – If sock and strm are collected in the same

sweep, sock.finalize may be called before strm.finalize


We Propose: “compensation stacks”

• Store cleanup code in run-time stacks– First-class objects, can pass them around

• When “action” succeeds, push “cleanup”– “action” and “cleanup” can be arbitrary code

• Pop all cleanup code and run it (LIFO) …– when the stack goes out of scope– or at end-of-scope if there is an exception– and/or early (when programmer specifies)– and when the stack is finalized

• Trace of events (loosely): – a1 a2 a3 a4 a5– a1 a2 … aX cX … c2 c1


Compensation Stacks

• Generalized destructors– No made-up objects needed for local cleanup– Can have multiple stacks

• e.g., one for each request in a webserver

• Annotate important interfaces to take compensation stacks– Cannot make a new socket without putting

“this.close()” on a stack of things-to-do– Everything on a stack will be run on all paths– Implicit “current scope” stack can optionally

be assumed by default


Where are we? We’re almost done!

Summary and Context Error Handling is Hard: Many Paths Finding Error-Handling Mistakes Characterizing Mistakes Old & New PL Features

• Case Studies• Grand Finale


Implementation, Case Studies

• Extend Java with such compensation stacks• Annotate key interfaces (socket, stream,

DB)• Annotate existing programs to use

compensation stacks– Both for library resources (easy)– And for “unique” cleanup actions– Add no new error handling– Ensure that existing handlers are run on all

paths

• Run programs on third-party workloads– Mark Brody, George Candea, Tom Martell, …


Case Study #1: Brown’s undo

• IMAP/SMTP Proxy (operator time travel)– Built for reliability

• 35,412 lines of Java, 128 annotation sites– Contains many unique cleanup actions

• 8- to 34-line code fragments

– As well as many standard “close” actions– And up to 5 simultaneous resources in

sequence

• Results– 225 lines shorter (~1%)– No performance changes

• Stack overhead dwarfed by I/O overhead


Case Study #2: Sun’s petstore

• “Amazon.com lite” e-commerce, inventory– Raises 150 exceptions over 3,900 requests– Avg Response Time: 52.06ms (std dev 100ms)

• 34,608 lines of Java, 123 annotation sites– Two hours of work– Standard “close” cleanup actions– 3 simultaneous resources: database handles

• Results– 168 lines shorter (~0.5%) – 0 such exceptions over 3,900 requests– Avg Response Time: 43.44ms (std dev 77ms)


Conclusion

• It is difficult to write error-handling code• We have an analysis for finding mistakes

– Dataflow analysis and fault model

• We found over 800 mistakes in 4 MLOC• We characterize these mistakes

– Programmers forget some paths

• Existing PL features are an awkward fit• We propose a new feature

– “compensation stacks” track your obligations

• And back it up with case studies


Any Questions?


False Negatives

• hibernate2 - if, return, if, if, FN#1• jatlite - if, if, if, if, field• quartz - FN#2, if, if, if, FN#3• ejbca - if, if, if, if, if• hsqldb - if, if, field, if, field• jboss - field, if, field, if, return• mckoi-sql - fld, fld, fld, ret, ret• osage - ret, if, if, ret, if• portal - ret, ret, ret, ret, ret• staf - field


False Negative #1 (ResultSet)

public Iterator iterate(Object[] values, Type[] types, RowSelection selection, Map namedParams, SessionImplementor session) throws HibernateException, SQLException { PreparedStatement st = prepareQueryStatement( getSQLString(), values,

types, selection, false, session ); try { bindNamedParameters(st, namedParams, session); setMaxRows(st, selection); ResultSet rs = st.executeQuery(); advance(rs, selection, session); return new IteratorImpl( rs, session, getReturnTypes(),

getScalarColumnNames() ); } catch (SQLException sqle) { JDBCExceptionReporter.logExceptions(sqle); closePreparedStatement(st, selection, session); throw sqle; } }


False Negative #2 (ObjectInputStream)

protected Object getObjectFromBlob(ResultSet rs, String colName) throws ClassNotFoundException, IOException, SQLException { Object obj = null;

Blob blobLocator = rs.getBlob(colName); InputStream binaryInput = null; try {

if (null != blobLocator && blobLocator.length() > 0) { binaryInput = blobLocator.getBinaryStream(); } } catch (Exception ignore) {} if (null != binaryInput) { ObjectInputStream in = new ObjectInputStream(binaryInput); obj = in.readObject(); in.close(); } return obj; }


False Negative #3 (ObjectInputStream)

protected Object getObjectFromBlob(ResultSet rs, String colName) throws ClassNotFoundException, IOException, SQLException { Object obj = null; InputStream binaryInput = rs.getBinaryStream(colName); if (binaryInput != null) { ObjectInputStream in = new ObjectInputStream(binaryInput); obj = in.readObject(); in.close(); }

return obj; }


What Are Those Benchmarks?

undo ROC group undoable email store and proxy

quartz Job scheduler for j2ee clustering transaction applications

infinity Browser and editor for Bioware’s Infinity Engine

ejbca Certificate authority (j2ee)

crm Web-based enterprise CRM workflow management portal

jogg MP3 Player

staf Software testing automation framework

hibernate2 Relational persistence for idiomatic java

jaxme XML binding framework for SAX2 (generates parsers)

hsqldb Relational database engine, jdbc support, SQL queries

cayenne Object relational persistence framework

jboss j2ee application server (“middleware OS”)

mckoi-sql Full-featured SQL’92 relational database

pcgen Character generator and maintenance program

compiere ERP+CRM solution for medium enterprise management


Show A Nested Try-Finally

// cayenne-1.0b4/src/…/dba/oracle/OraclePkGenerator.java, line 254protected List getExistingSequences(DataNode node) throws SQLException { // check existing sequences Connection con = node.getDataSource().getConnection(); try { Statement sel = con.createStatement(); try { ResultSet rs = sel.executeQuery("SELECT LOWER(SEQUENCE_NAME) FROM

ALL_SEQUENCES"); try { List sequenceList = new ArrayList(); while (rs.next()) sequenceList.add(rs.getString(1)); return sequenceList; } finally { rs.close(); } } finally { sel.close(); } } finally { con.close(); }} Most nested try-finally cases are not

this easy to follow.


Show Us Another Bug

Note error handler that prints a stack traceand does nothing else ...

// com/ohioedge/j2ee/api/org/tool/ejb/LetterTemplateEJB.java, line 154private StringBuffer getColumnStringBufferFromReader(){ StringBuffer buf = null; java.io.Reader reader = null; PreparedStatement prepStmt = null; ResultSet rs = null; Connection cn = null; try { cn = ConnectionFactory.getConnection("jdbc/LetterTemplateDB"); StringBuffer qry = new StringBuffer(); qry.append(" SELECT \"letterTemplate\" FROM LetterTemplate "); qry.append(" WHERE \"letterTemplateID\" = "+getLetterTemplateID()); prepStmt = cn.prepareStatement(qry.toString()); rs = prepStmt.executeQuery(); while (rs.next()) reader = rs.getCharacterStream(1); buf = readIntoStringBuffer(reader); rs.close(); prepStmt.close(); } catch (Exception e) { log.error(this.getClass().getName()+".getColumnStringBufferFromReader():"); e.printStackTrace(); } finally { try { cn.close(); } catch (Exception e1) { log.error(this.getClass().getName()+".getColumnStringBufferFromReader():"); e1.printStackTrace(); } } return buf; }


Show A False Positive

// org/axiondb/tools/Console.java, line 89try { _conn = DriverManager.getConnection(buf.toString()); _stmt = _conn.createStatement();} catch (SQLException e) { cleanUp(); throw e;}// …public void cleanUp() { try { _rset.close(); } catch (Exception e) {} try { _stmt.close(); } catch (Exception e) {} try { _conn.close(); } catch (Exception e) {}}

finding and preventing run-time error handling mistakes wes weimer george necula uc berkeley...

Documents