testing concurrent programs

Testing Concurrent ProgramsTesting Concurrent Programs

Rice Computer Science ClubRice Computer Science ClubMathias RickenMathias RickenRice UniversityRice UniversityOctober 4, 2007October 4, 2007

Moore’s LawMoore’s Law

TimelinessTimeliness

CPU clock frequencies stagnateCPU clock frequencies stagnate

Multi-Core CPUs provide additional Multi-Core CPUs provide additional processing powerprocessing power– Multiple threads needed to use multiple coresMultiple threads needed to use multiple cores

Writing concurrent programs is difficult!Writing concurrent programs is difficult!

Programming ExamplesProgramming Examples

Unit TestingUnit Testing

Unit tests…Unit tests…– Test a part, not the whole programTest a part, not the whole program– Occur earlierOccur earlier– Automate testingAutomate testing– Serve as documentationServe as documentation– Prevent bugs from reoccurringPrevent bugs from reoccurring– Help keep the shared repository cleanHelp keep the shared repository clean

Effective with a single thread of controlEffective with a single thread of control

Foundation of Unit TestingFoundation of Unit Testing

Unit tests depend on deterministic Unit tests depend on deterministic behaviorbehaviorKnown input, expected output…Known input, expected output…

SuccessSuccess correct behaviorcorrect behaviorFailureFailure flawed codeflawed code

Outcome of test is meaningfulOutcome of test is meaningful

Problems Due to ConcurrencyProblems Due to ConcurrencyThread scheduling is nondeterministic and Thread scheduling is nondeterministic and machine-dependentmachine-dependent– Code may be executed under different schedulesCode may be executed under different schedules– Different schedules may produce different resultsDifferent schedules may produce different results

Known input, expected output…Known input, expected output…

SuccessSuccess correct behaviorcorrect behavior in this schedulein this schedule, , may be may be flawedflawed in other schedule in other schedule

FailureFailure flawed codeflawed code

Success of unit test is meaninglessSuccess of unit test is meaningless

Possible SolutionsPossible Solutions

Programming Language FeaturesProgramming Language Features– Ensuring that bad things cannot happenEnsuring that bad things cannot happen– May restrict programmersMay restrict programmersLock-Free AlgorithmsLock-Free Algorithms– Ensuring that if bad things happen, it’s okEnsuring that if bad things happen, it’s ok– May limit data structures availableMay limit data structures availableComprehensive TestingComprehensive Testing– Testing if bad things happen in any scheduleTesting if bad things happen in any schedule– Does not prevent problems, but does not limit Does not prevent problems, but does not limit

solutions eithersolutions either

ContributionsContributions

Improvements to JUnitImprovements to JUnit– Detect exceptions and failed assertions in Detect exceptions and failed assertions in

threads other than the main threadthreads other than the main threadAnnotations for Concurrency InvariantsAnnotations for Concurrency Invariants– Express complicated requirements about Express complicated requirements about

locks and threadslocks and threadsTools for Schedule-Based ExecutionTools for Schedule-Based Execution– Record, deadlock monitorRecord, deadlock monitor– Random delays, random yieldsRandom delays, random yields

Improvements to JUnitImprovements to JUnit

Uncaught exceptions and failed assertionsUncaught exceptions and failed assertions– Not caught in child threadsNot caught in child threads

Sample JUnit TestsSample JUnit Testspublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} public void public void testAssertion() {testAssertion() { assertEquals(0, 1);assertEquals(0, 1); }}}}

if (0!=1) throw new AssertionFailedError();

}}Both tests

fail.Both tests

fail.

Problematic JUnit TestsProblematic JUnit Testspublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}

newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }}}).start();}).start();

throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!");

Main thread

Child thread

Main thread

Child thread

spawns

uncaught!

end of test

success!

Problematic JUnit TestsProblematic JUnit Testspublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}



Main thread

Child threadUncaught exception,

test should fail but does not!


Uncaught exceptions and failed assertionsUncaught exceptions and failed assertions– Not caught in child threadsNot caught in child threads

Thread group with exception handlerThread group with exception handler– JUnit test runs in a separate thread, not main threadJUnit test runs in a separate thread, not main thread– Child threads are created in same thread groupChild threads are created in same thread group– When test ends, check if handler was invokedWhen test ends, check if handler was invoked

Thread Group for JUnit TestsThread Group for JUnit Testspublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}



Test thread

Child thread

invokeschecks

TestGroup’s Uncaught Exception Handler

Thread Group for JUnit TestsThread Group for JUnit Testspublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}



Test thread

Child thread

Test thread

Child thread

spawns

uncaught!

end of testfailure!

invokes group’s handler

Main thread

spawns and waits resumes

check group’s handler

Improvements to JUnitImprovements to JUnitUncaught exceptions and failed assertionsUncaught exceptions and failed assertions– Not caught in child threadsNot caught in child threads

Thread group with exception handlerThread group with exception handler– JUnit test runs in a separate thread, not main threadJUnit test runs in a separate thread, not main thread– Child threads are created in same thread groupChild threads are created in same thread group– When test ends, check if handler was invokedWhen test ends, check if handler was invoked

Detection of uncaught exceptions and failed Detection of uncaught exceptions and failed assertions in child threads that occurred before assertions in child threads that occurred before test’s endtest’s end

Past tense: occurred!

Child Thread Outlives ParentChild Thread Outlives Parentpublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}



Test thread

Child thread

Test thread

Child thread

spawns

uncaught!

end of testfailure!


Main thread

spawns and waits resumes

check group’s handler

Child Thread Outlives ParentChild Thread Outlives Parentpublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} }).start();}).start(); }}}}



Test thread

Child thread

Test thread

Child thread

spawns

uncaught!end of test

success!


Main thread

spawns and waits resumescheck group’s

handler

Too late!


Child threads are not required to terminateChild threads are not required to terminate– A test may pass before an error is reachedA test may pass before an error is reached

Detect if any child threads are still aliveDetect if any child threads are still alive– Declare failure if test thread has not waitedDeclare failure if test thread has not waited– Ignore daemon threads, system threads (AWT, RMI, Ignore daemon threads, system threads (AWT, RMI,

garbage collection, etc.)garbage collection, etc.)

Previous schedule is a test failurePrevious schedule is a test failure– Should be prevented by using Should be prevented by using Thread.join()Thread.join()

Enforced JoinEnforced Joinpublicpublic class class Test Test extends extends TestCase {TestCase { public void public void testException() {testException() { newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }} });}); t.start(); … t.join();t.start(); … t.join(); }}}}

Thread t = Thread t = newnew Thread(new Runnable() { Thread(new Runnable() { public void run() {public void run() { throwthrow newnew RuntimeException("booh!"); RuntimeException("booh!"); }}});});t.start(); … t.join(); …t.start(); … t.join(); …


Test thread

Child thread


Child threads are not required to terminateChild threads are not required to terminate– A test may pass before an error is reachedA test may pass before an error is reached

Detect if any child threads are still aliveDetect if any child threads are still alive– Declare failure if test thread has not waitedDeclare failure if test thread has not waited– Ignore daemon threads, system threads (AWT, RMI, Ignore daemon threads, system threads (AWT, RMI,

garbage collection, etc.)garbage collection, etc.)

Previous schedule is a test failurePrevious schedule is a test failure– Should be prevented by using Should be prevented by using Thread.join()Thread.join()

Testing ConcJUnitTesting ConcJUnitReplacement for junit.jar or as plugin JAR for Replacement for junit.jar or as plugin JAR for JUnit 4.2JUnit 4.2

Available as binary and source at Available as binary and source at http://www.concutest.org/http://www.concutest.org/

Results from DrJava’s unit testsResults from DrJava’s unit tests– Child thread for communication with slave VM still Child thread for communication with slave VM still

alive in testalive in test– Several reader and writer threads still alive in low Several reader and writer threads still alive in low

level test (calls to level test (calls to join()join() missing) missing)

http://www.concutest.org/

ConclusionConclusion

Improved JUnit now detects problems in Improved JUnit now detects problems in other threadsother threads– Only in chosen scheduleOnly in chosen schedule– Needs schedule-based executionNeeds schedule-based executionAnnotations ease documentation and Annotations ease documentation and checking of concurrency invariantschecking of concurrency invariants– Open-source library of Java API invariantsOpen-source library of Java API invariantsSupport programs for schedule-based Support programs for schedule-based executionexecution

Future WorkFuture Work

Schedule-Based ExecutionSchedule-Based Execution– Replay given scheduleReplay given schedule– Generate possible schedulesGenerate possible schedules– Dynamic race detectionDynamic race detection– Probabilities/durations for random Probabilities/durations for random

yields/sleepsyields/sleepsExtend annotations to Floyd-Hoare logicExtend annotations to Floyd-Hoare logic– Preconditions, postconditionsPreconditions, postconditions– Representation invariantsRepresentation invariants

Many Thanks To…Many Thanks To…

My advisorMy advisor– Corky CartwrightCorky Cartwright

My committee membersMy committee members– Walid TahaWalid Taha– Bill SchererBill Scherer

NFS and Texas ATPNFS and Texas ATP– For partially providing fundingFor partially providing funding

Rice Computer Science ClubRice Computer Science Club

Extra SlidesExtra Slides

Test all possible schedulesTest all possible schedules– Concurrent unit tests meaningful againConcurrent unit tests meaningful again

Number of schedules (Number of schedules (NN))– tt: # of threads, : # of threads, ss: # of slices per thread: # of slices per thread

detail

Tractability of Comprehensive TestingTractability of Comprehensive Testing

Extra: Number of SchedulesExtra: Number of Schedules

back

Product of s-combinations

For thread 1: choose s out of ts time slicesFor thread 2: choose s out of ts-s time slices…For thread t-1: choose s out of 2s time slicesFor thread t-1: choose s out of s time slices

Writing s-combinations using factorial

Cancel out terms in denominator and next numerator

Left with (ts)! in numerator and t numerators with s!

If program is race-free, we do not have to If program is race-free, we do not have to simulate all thread switchessimulate all thread switches– Threads interfere only at “critical points”: lock Threads interfere only at “critical points”: lock

operations, shared or volatile variables, etc.operations, shared or volatile variables, etc.– Code between critical points cannot affect outcomeCode between critical points cannot affect outcome– Simulate all possible arrangements of blocks Simulate all possible arrangements of blocks

delimited by critical pointsdelimited by critical points

Run dynamic race detection in parallelRun dynamic race detection in parallel– Lockset algorithm (e.g. Eraser by Savage et al)Lockset algorithm (e.g. Eraser by Savage et al)

Tractability of Comprehensive TestingTractability of Comprehensive Testing

Critical Points ExampleCritical Points Example

Thread 1

Thread 2

Local Var 1

Local Var 1

Shared Var

Lock

lock access unlock

lock access unlock

lock access unlock

All accesses protected by

lock

Local variables don’t need

locking


lock


lock

Fewer critical points than thread switchesFewer critical points than thread switches– Reduces number of schedulesReduces number of schedules– Example:Example: Two threads, but no communicationTwo threads, but no communication

NN = 1 = 1

Unit tests are smallUnit tests are small– Reduces number of schedulesReduces number of schedules

Hopefully comprehensive simulation is tractableHopefully comprehensive simulation is tractable– If not, heuristics are still better than nothingIf not, heuristics are still better than nothing

Fewer SchedulesFewer Schedules

LimitationsLimitationsImprovements only check chosen Improvements only check chosen scheduleschedule– A different schedule may still failA different schedule may still fail– Requires comprehensive testing to be Requires comprehensive testing to be

meaningfulmeaningfulMay still miss uncaught exceptionsMay still miss uncaught exceptions– Specify absolute parent thread group, not Specify absolute parent thread group, not

relativerelative– Cannot detect uncaught exceptions in a Cannot detect uncaught exceptions in a

program’s uncaught exception handler (JLS program’s uncaught exception handler (JLS limitation)limitation)

details

Extra: LimitationsExtra: Limitations

May still miss uncaught exceptionsMay still miss uncaught exceptions– Specify absolute parent thread group, not Specify absolute parent thread group, not

relative (rare)relative (rare)Koders.com: 913 matches Koders.com: 913 matches ThreadGroupThreadGroup vs. vs. 49,329 matches for 49,329 matches for ThreadThread

– Cannot detect uncaught exceptions in a Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS program’s uncaught exception handler (JLS limitation)limitation)

Koders.com: 32 method definitions for Koders.com: 32 method definitions for uncaughtExceptionuncaughtException method method

back

Extra: DrJava StatisticsExtra: DrJava Statistics20042004

73673661061036369090

511651164161416196596518.83%18.83%

10710711

Unit testsUnit testspassedpassedfailedfailednot runnot run

InvariantsInvariantsmetmetfailedfailed% failed% failed

KLOCKLOC““event thread”event thread”

20062006881881

8818810000

344123441230616306163796379611.0311.03

1291299999

back

Concurrency InvariantsConcurrency Invariants

Has to be called in event threadHas to be called in event thread– TableModelTableModel, , TreeModelTreeModel

May not be called in event threadMay not be called in event thread– invokeAndWait()invokeAndWait()

Have to acquire readers/writers lockHave to acquire readers/writers lock– AbstractDocumentAbstractDocument– DrJava’s documentsDrJava’s documents

Invariants Difficult to DetermineInvariants Difficult to Determine

May be found inMay be found in– Javadoc commentsJavadoc comments– Only in internal commentsOnly in internal comments– WhitepapersWhitepapers

Often not documented at allOften not documented at allErrors not immediately evidentErrors not immediately evidentImpossible to check automaticallyImpossible to check automatically

Java AnnotationsJava Annotations

Add invariants as annotationsAdd invariants as annotations

@NotEventThread@NotEventThreadpublic static void invokeAndWait(public static void invokeAndWait( Runnable r) { … } Runnable r) { … }

Process class filesProcess class files– Find uses of annotationsFind uses of annotations– Insert bytecode to check invariants at method Insert bytecode to check invariants at method

beginningbeginning

Advantages of AnnotationsAdvantages of AnnotationsJava Language constructsJava Language constructs– Syntax checked by compilerSyntax checked by compiler

Easy to apply to part of the programEasy to apply to part of the program– e.g. when compared to a type system changee.g. when compared to a type system change

Light-weightLight-weight– Negligible runtime impact if not debugging (slightly Negligible runtime impact if not debugging (slightly

bigger class files)bigger class files)

Automatic CheckingAutomatic Checking

Predicate AnnotationsPredicate Annotations

In annotation definition, specify In annotation definition, specify static static booleanboolean Java method Java method– Method must be callable from every contextMethod must be callable from every context

completely static and public completely static and public

Data in annotation, method arguments and Data in annotation, method arguments and value of value of thisthis passed when method passed when method invokedinvoked

Predicate Annotation ExamplePredicate Annotation Example@PredicateLink(value=Predicates.class,@PredicateLink(value=Predicates.class, method="example",method="example", arguments=true)arguments=true)publicpublic @ @interfaceinterface ExampleAnnotation { ExampleAnnotation { String foo;String foo;}}

Definition

Refers to Predicates.example

Predicate Annotation ExamplePredicate Annotation Examplepublic class public class TestCode {TestCode { @ExampleAnnotation(foo="test")@ExampleAnnotation(foo="test") public void public void test(test(intint param) { … } param) { … }}}……TestCode t = TestCode t = newnew TestCode(); t.test(5); TestCode(); t.test(5);

Usage

Call

Predicate Annotation ExamplePredicate Annotation Examplepublic class public class Predicates {Predicates { public static boolean public static boolean example(example( Object this0,Object this0, intint param, param, String foo) {String foo) { returnreturn (foo.length()<param); (foo.length()<param);}}

Predicate Annotation ExamplePredicate Annotation Examplepublic class public class TestCode {TestCode { @ExampleAnnotation(foo="@ExampleAnnotation(foo="testtest")") public void public void test(test(intint param){…} param){…}}}……TestCode TestCode tt = = newnew TestCode(); TestCode();tt.test(.test(55););

@PredicateLink(value=@PredicateLink(value=PredicatesPredicates.class,.class, method="method="exampleexample",", arguments=true)arguments=true)publicpublic @ @interfaceinterface ExampleAnnotation { ExampleAnnotation { String foo;String foo;}}

public class public class PredicatesPredicates { { public static boolean public static boolean exampleexample(( Object Object this0this0,, intint paramparam,, String String foofoo) {) { returnreturn (foo.length()<param); // (foo.length()<param); // this0==tthis0==t, , param==5param==5, , foo=="test"foo=="test"}}

Invariant Annotation LibraryInvariant Annotation Library

@OnlyEventThread@OnlyEventThread, , @NotEventThread@NotEventThread@OnlyThreadWithName@OnlyThreadWithName@NotNullArgument@NotNullArgument@DistinctArguments@DistinctArguments, , @SameArguments@SameArguments@OnlySynchronizedThis@OnlySynchronizedThis, , @NotSynchronizedThis@NotSynchronizedThis@OnlySynchronizedArgument@OnlySynchronizedArgument,, @NotSynchronizedArgument@NotSynchronizedArgumentetc. (ca. 80 annotations)etc. (ca. 80 annotations)

Problem: Multiple AnnotationsProblem: Multiple Annotations

Java does not allow the same annotation Java does not allow the same annotation class multiple timesclass multiple times

@@OnlyThreadWithNameOnlyThreadWithName("foo")("foo")@@OnlyThreadWithNameOnlyThreadWithName("bar") // error("bar") // errorvoid void testMethod() { … }testMethod() { … }

Conjunctions, disjunctions and negations?Conjunctions, disjunctions and negations?

Annotation Subclasses?Annotation Subclasses?

Let annotation extend a supertype?Let annotation extend a supertype?

publicpublic @ @interfaceinterface Invariant { } Invariant { }publicpublic @ @interfaceinterface OnlyThreadWithName OnlyThreadWithName extendsextends Invariant { String name(); } Invariant { String name(); }publicpublic @ @interfaceinterface And And extendsextends Invariant { Invariant { Invariant[] terms();Invariant[] terms();}}

Subtyping not allowed for annotationsSubtyping not allowed for annotations

Generic Annotations?Generic Annotations?

Write Write @And@And as generic annotation? as generic annotation?

publicpublic @ @interfaceinterface And<T> { And<T> { T[] terms();T[] terms();}}publicpublic @ @interfaceinterface OnlyThreadWithName { OnlyThreadWithName { String name();String name();}}

Generics not allowed in annotationsGenerics not allowed in annotations

Work-AroundWork-Around

Different meta-annotation, Different meta-annotation, @Combine@Combine

@Combine@Combine(Combine.Mode.AND)(Combine.Mode.AND)publicpublic @ @interfaceinterface SeveralNames { SeveralNames { OnlyThreadWithName[] value();OnlyThreadWithName[] value();}}

@SeveralNames({@OnlyThreadWithName("foo"),@SeveralNames({@OnlyThreadWithName("foo"), @OnlyThreadWithName("bar")})@OnlyThreadWithName("bar")})void void testMethod() { … }testMethod() { … }

Combine AnnotationsCombine Annotations

May only contain invariant annotationsMay only contain invariant annotations– Predicate annotationsPredicate annotations– Combine annotationsCombine annotations– Arrays of the aboveArrays of the above

Predicate method automatically generatedPredicate method automatically generated– Calls member predicate methodsCalls member predicate methods– Accumulates using AND, OR or NOTAccumulates using AND, OR or NOT

NOT first negates, then uses ANDNOT first negates, then uses ANDDefault mode is ORDefault mode is ORDe Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)

Invariant InheritanceInvariant InheritanceInvariants on a methodInvariants on a method– Apply to the method and all overriding methods in Apply to the method and all overriding methods in

subclassessubclassesInvariants on a classInvariants on a class– Apply to all methods introduced in that class or Apply to all methods introduced in that class or

subclassessubclasses

Methods can have invariants defined elsewhereMethods can have invariants defined elsewhere

All annotations describe requirements for the All annotations describe requirements for the client (and, due to subclassing, for subclasses)client (and, due to subclassing, for subclasses)– Allows frameworks to describe requirementsAllows frameworks to describe requirements– Description “thread-safe” is often wrongDescription “thread-safe” is often wrong

Invariant SubtypingInvariant SubtypingTo maintain substitutability, subclasses may not To maintain substitutability, subclasses may not strengthen invariantsstrengthen invariants

Invariants can be modeled as special input Invariants can be modeled as special input parameterparameter– Tuple of invariants (“record” in λ calculus [Pierce])Tuple of invariants (“record” in λ calculus [Pierce])– Subtyping rules for records declare the “wider” record Subtyping rules for records declare the “wider” record

as subtypeas subtype– In function types, parameter types are contravariant In function types, parameter types are contravariant

II00 = {}, I = {}, I11 = {inv = {inv11}, I}, I22 = {inv = {inv11,inv,inv22}, I}, I22 <: I <: I11 <: I <: I00

FF0 = 0 = II00 →→ ·, F ·, F1 = 1 = II11 →→ ·, F ·, F2 = 2 = II22 →→ ·, F ·, F00 <: F <: F11 <: F <: F22

Invariant SubtypingInvariant SubtypingAnalyze methods with invariants as parameterAnalyze methods with invariants as parameter

Invariants subtyping: A Invariants subtyping: A <@<@ B B <@<@ C C

IIAA = {}, I = {}, IBB = {inv = {inv11}, I}, ICC = {inv = {inv11,inv,inv22}; I}; ICC <: I <: IBB <: I <: IAA

FFA = A = IIAA →→ ·, F ·, FB = B = IIBB →→ ·, F ·, FC = C = IICC →→ ·; F ·; FAA <: F <: FBB <: F <: FCC

Java subtyping:Java subtyping: C <: B <: A C <: B <: A

classclass A { A {

voidvoid f() f() { … };{ … };}}

classclass B B extendsextends A { A { @Inv1@Inv1 voidvoid f() f() { … };{ … };}}

classclass C C extendsextends B { B { @Inv2@Inv2 voidvoid f() f() { … };{ … };}}

Detection of Subtyping ProblemsDetection of Subtyping Problems

If Java subtyping and invariant subtyping If Java subtyping and invariant subtyping disagree (A <: B but B disagree (A <: B but B <@<@ A) A)– Substitutability not maintainedSubstitutability not maintained– Statically emit warningStatically emit warning

Detect if client subclasses do not use Detect if client subclasses do not use framework classes as prescribedframework classes as prescribed– Safer multithreaded frameworksSafer multithreaded frameworks

Java API AnnotationsJava API Annotations

Started to annotate methods in Java APIStarted to annotate methods in Java API– 30 whole classes, 44 individual methods30 whole classes, 44 individual methods

Community project at Community project at http://community.http://community.concutestconcutest.org/.org/– Anyone can suggest annotationsAnyone can suggest annotations– Vote on suggested annotationsVote on suggested annotations– Browse by class or annotation typeBrowse by class or annotation type

Annotations can be extracted into XMLAnnotations can be extracted into XML– Share annotationsShare annotations– Add checks without needing source codeAdd checks without needing source code

http://community.concutest.org/



Testing Invariant CheckerTesting Invariant Checker

Annotated two DrJava versionsAnnotated two DrJava versions– 3/26/20043/26/2004– 9/2/20069/2/2006

Ran test suite, logged invariant violationsRan test suite, logged invariant violations– 2004: 18.83% failed2004: 18.83% failed– 2006: 11.03% failed2006: 11.03% failed

2006 version easier to annotate2006 version easier to annotate– Better documentation of invariantsBetter documentation of invariants

testing concurrent programs

Documents