the closures controversy joshua bloch chief java architect google inc
TRANSCRIPT
www.javapolis.com
Disclaimer: Talk Represents My Opinion, Not Google’s!
Google believes that the Java platform would likely benefit from some additional support for closures. Like the broader Java community, we are divided on what form this support should take. Some favor a lightweight approach that addresses the pain of anonymous inner classes without affecting the type system, VM, or libraries; others favor a heavyweight approach designed to provide for programmer-defined control structures. We believe it is premature to launch a JSR that forces us down either path.
www.javapolis.com
Outline
I. Setting the Stage
II. Why Enhance Support for Closures?
III. BGGA Closures
IV. A Lightweight Approach
V. Where Do We Go From here?
OOPSLA Invited Talk
October 8, 1996
Digitally reconstructed
from the Internet Archive
(AKA the Wayback Machine)
by Joshua Bloch
November 24, 2007
Oak
• Started as a reimplementation of C++
• Always a tool, never an end itself
• Took on a life of its own
• The Web happened...• serendipitous match!
• and it became Java
The Java Language
Fusion of four kinds of programming
• Object Oriented like Simula/C++/ ObjectiveC…
• Numeric like FORTRAN
• Systems like C
• Distributed like nothing else
Java - a language for a job
• Doing language research:an anti-goal
• Started using C++
• Broke down, needed:• Architecture neutral, portable, reliable, safe,
long lived, multithreaded, dynamic, simple, ...
Practical, not theoretical
• Driven by what people needed• (but hey, I spent too much time going to school!)
• Theory provides• rigour• cleanliness• cohesiveness
No new ideas here
• Shamelessly ripped off ideas that worked in C, C++, Objective C, Cedar/Mesa, Modula, Simula, ...
• (well, we slipped once or twice and invented something)
Don’t fix it until it chafes
• To keep it simple...
• A procedural principle:• Require several real instances before
including a feature• i.e. nothing goes in because it’s “nice”
• (== “Just Say No, until threatened with bodily harm”)
Java feels...
• Hyped :-(
• Playful
• Flexible
• Deterministic
• Non-threatening
• Rich
• Like I can just write code…
Hey!
I
left
ou
t “O
bje
ct-
Ori
en
ted
”!
www.javapolis.com
So How Are We Doing? Not So Well, Unfortunately
Enum<E extends Enum<E>> { ... }
<T extends Object & Comparable<? super T>> T Collections.max(Collection<? extends T>)
public <V extends Wrapper<? extends Comparable<T>>>
Comparator<V> comparator() { ... }
error: equalTo(Box<capture of ?>) in Box<capture of ?> cannot be applied to (Box<capture of ?>)
equal = unknownBox.equalTo(unknownBox)
Arrays.asList(String.class, Integer.class) // Warning!
See Angelia Langer's 427-page (!) Java Generics FAQ for more:
http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.pdf
www.javapolis.com
What the Man on the Web is Saying
“I am completely and totally humbled. Laid low. I realize now that I am simply not smart at all. I made the mistake of thinking that I could understand generics. I simply cannot. I just can't. This is really depressing. It is the first time that I've ever not been able to understand something related to computers, in any domain, anywhere, period.”
“I'm the lead architect here, have a PhD in physics, and have been working daily in Java for 10 years and know it pretty well. The other guy is a very senior enterprise developer (wrote an email system that sends 600 million emails/year with almost no maintenance). If we can't get [generics], it's highly unlikely that the ‘average’ developer will ever in our lifetimes be able to figure this stuff out.”
www.javapolis.com
Where Does The Complexity Come From?
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10
Features
Co
mp
lexi
ty
Features(linear)
Feature Pairs(quadratic)
Feature Tuples(exponential)
www.javapolis.com
If The Feel of Java is to be Preserved...
We simply cannot afford another wildcards Further language additions must be
undertaken with extreme caution Minimal addition to conceptual surface area High power-to-weight ratio
www.javapolis.com
Outline
I. Setting the Stage
II. Why Enhance Support for Closures?
II. BGGA Closures
III. A Lightweight Approach
IV. Where Do We Go From here?
www.javapolis.com
What is a Closure?
One definition: a function that is evaluated in an environment containing one or more bound variables [Wikipedia]
In English: a little snippet of code that can be passed around for subsequent execution
Limited support for closures since JDK 1.1, in the form of anonymous classes
www.javapolis.com
Why Are We Considering Better Support for Closures?
Fine-Grained Concurrecy - Passing snippets of code to fork-join frameworks using anonymous classes is a pain
Resource Managememnt - Using try-finally blocks for resource management is a pain, and causes resource leaks
www.javapolis.com
1. Fine-grained (fork-join) concurrency
"It has to be easier to send snippets of code to frameworks for parallel execution; otherwise no one will use them .“
–Doug Lea, 2005
www.javapolis.com
Here’s How it Looks Today
class StudentStatistics { ParallelArray<Student> students = ... // ... public double getMaxSeniorGpa() { return students.withFilter(isSenior). withMapping(gpaField).max(); }
// helpers: static final class IsSenior implements Predicate<Student> { public boolean evaluate(Student s) { return s.credits > 90; } } static final IsSenior isSenior = new IsSenior(); static final class GpaField implements MapperToDouble<Student> { public double map(Student s) { return s.gpa; } } static final GpaField gpaField = new GpaField(); }
www.javapolis.com
2. Automatic Resource Management
“The C++ destructor model is exactly the same as the Dispose pattern, except that it is far easier to use and a direct language feature and correct by default, instead of a coding pattern that is off by default and causing correctness or performance problems when it is forgotten.”
–Herb Sutter, OOPSLA 2004
www.javapolis.com
How it Looks Today–Manual Resource Management
static String readFirstLineFromFile(String path) throws IOException { BufferedReader r = null; String s; try { r = new BufferedReader(new FileReader(path)); s = r.readLine(); } finally { if (r != null) r.close(); } return s;}
www.javapolis.com
It’s Worse With Multiple Resources (Puzzler 41)
static void copy(String src, String dest) throws IOException { InputStream in = null; OutputStream out = null; try { in = new FileInputStream(src); out = new FileOutputStream(dest); byte[] buf = new byte[1024]; int n; while ((n = in.read(buf)) >= 0) out.write(buf, 0, n); } finally { closeIgnoringException(in); closeIgnoringException(out); }}
private static void closeIgnoringException(Closeable c) { if (c != null) { try { c.close(); } catch (IOException ex) { // ignore } }}
www.javapolis.com
Automatic Resource Management
C++ DestructorString^ ReadFirstLineFromFile( String^ path ) {
StreamReader r(path);
return r.ReadLine();
}
C# using BlockString ReadFirstLineFromFile( String path ) {
using ( StreamReader r = new StreamReader(path) ) {
return r.ReadLine();
}
}
www.javapolis.com
Outline
I. Setting the Stage
II. Why Enhance Support for Closures?
III. BGGA Closures
IV. A Lightweight Approach
V. Where Do We Go From here?
www.javapolis.com
Controversial Features in BGGA
Function types Non-local return Non-local break and continue Unrestricted access to nonfinal local variables Design goal: library-defined control constructs
www.javapolis.com
Function Types
The BGGA Spec says: “While the subtype rules for function types may at first glance appear arcane, they are defined this way for very good reason: [...]. And while the rules seem complex, function types do not add complexity to Java's type system because function types and their subtype relations can be understood as a straightforward application of generics and wildcards (existing constructs). From the programmer's perspective, function types just ‘do the right thing.’”
www.javapolis.com
Function Types are Hard to Read
static Pair<{ => int },{ => int }> joinedCounters(int initial) { return Pair.<{ => int },{ => int }>of( { => initial++ }, { => initial++ });}
interface BThunk extends {=>boolean} { }static final {BThunk, { => void} => void} wihle = {BThunk cond, { => void } action => while (cond.invoke()) action.invoke(); };
static <throws X> { {=> void throws X} => void throws X }foo() { return { { => void throws X } block => block.invoke(); };}
These examples come from test code that ships with BGGA Prototype
www.javapolis.com
Function Types Encourage an “Exotic” Programming Style
static <A1, A2, R> {A1 => {A2 => R}} curry({A1, A2 => R} fn) { return {A1 a1 => {A2 a2 => fn.invoke(a1, a2)}};} <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1); static <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1) { return {A2 a2, A3 a3 => fn.invoke(a1, a2, a3)};}
From Mark Mahieu's blog:Currying and Partial Application with Java Closureshttp://markmahieu.blogspot.com/2007/12/currying-and-partial-
application-with.html
www.javapolis.com
Nominal Types are Rich Compared to Function Types
{T, T => T}
Name →
Known Implementations →
Documentation, including semantic constraints
www.javapolis.com
Function Types Have Unexpected Interactions
Arrays don’t work Foo.java:6: generic array creation { => int}[] closures = new { => int}[N];
Autoboxing doesn’t work LoopBenchC.java:10: <E,X>forEach(java.util.Collection<E>,
{E => void throws X}) in LoopBenchC cannot be applied to (java.util.List<java.lang.Integer>,{int => void})
forEach(list, {int i =>
Wildcards produce difficult error messages NewtonWithClosures.java:26: invoke(capture#418 of ? super
{double => double}) in {capture#418 of ? super {double => double} => capture#928 of ? extends {double => double}} cannot be applied to (double)
return fixedPoint(transform.invoke(guess)); ^
www.javapolis.com
Function Types Limit Interoperability With SAM Types
Closure conversion only works with interfaces
Unfortunately, existing APIs sometimes use
abstract classes for functions e.g., TimerTask, SwingWorker
These APIs would become 2nd class citizens
www.javapolis.com
Summary - Pros and Cons of Function Types
+ Avoid need to define named interfaces+ Avoid incompatibility among SAM types with
same signatures- Hard to read under moderate-to-heavy use- Encourage “exotic” style of programming- Don't reflect semantic constraints- Don't provide the same level of documentation- Don’t interact well with autocompletion (or grep)- Limited interoperability may balkanize libraries
www.javapolis.com
BGGA Closures Have Two Kinds of Returns
static boolean test(boolean arg) { {boolean => boolean} closure = { boolean arg => if (arg) return true; // Non-local return false // local return }; return !closure.invoke(arg);}
return means something completely different in a BGGA closure and an anonymous class
www.javapolis.com
What Does test() Return?
static <E> Boolean contains(Iterable<E> seq, Predicate<E> pred) { for (E e : seq) if (pred.invoke(e)) return true; return false;}
static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, new Predicate<Character>() { public Boolean invoke(Character c) { return c > 'j'; } });}
interface Predicate<T> { Boolean invoke(T t); }
www.javapolis.com
Now What Does test() Return? (BGGA)
static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false;}
static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; } );}
www.javapolis.com
Now What Does test() Return? (BGGA)
static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false;}
static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; });}
Accidental non-local return due to cut-and-paste from anonymous class can cause insidious bug
www.javapolis.com
Suppose You Wanted to Translate this Method to BGGA
static <E> Predicate<Iterable<E>> contains( final Predicate<E> pred) { return new Predicate<Iterable<E>>() { public Boolean invoke(Iterable<E> seq) { for (E e : seq) if (pred.invoke(e)) return true; return false; } };}
www.javapolis.com
It’s Awkward, as Only One Local Return is Permitted
static <E> { Iterable<E> => Boolean } contains( { E => Boolean } pred) { return { Iterable<E> seq => Boolean result = false; for (E e : seq) { if (pred.invoke(e)) { result = true; break; } } result };}
www.javapolis.com
Summary - Pros and Cons of Non-Local Returns
+ Permits library-defined control structures
- Having two kinds of returns is confusing
- Meaning of return has changed
- Unintentional non-local returns can cause bugs
- Only one local return permitted per closure
www.javapolis.com
What Does This Program Print?
public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>();
for (int i = 0; i < N; i++) closures.add( { => i } );
int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); }}
www.javapolis.com
What does this program print?
public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>();
for (int i = 0; i < N; i++) closures.add( { => i } );
int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); }}
It prints 100, not 45. The same loop variable is captured by all 10 closures and evaluated after the loop is finished!
www.javapolis.com
Summary - Pros and Cons of Access to Nonfinal Locals
+ Permits library-defined control structures
+ Eliminates some uses of final modifier
- Semantics can be very confusing- Locals can persist after their scope is finished
- Locals can be modified by other threads
- Performance model for locals will change
www.javapolis.com
Library-Defined Control Constructs
What are the compelling use-cases? Custom loops Automatic resource management blocks Timer block
www.javapolis.com
Custom for-loops (Example from BGGA Spec)
<K,V,throws X>void for eachEntry(Map<K,V> map, {K,V=>void throws X} block) throws X { for (Map.Entry<K,V> entry : map.entrySet()) { block.invoke(entry.getKey(), entry.getValue()); }}
for eachEntry(String name, Integer value : map) { if ("end".equals(name)) break; if (name.startsWith("com.sun.")) continue; System.out.println(name + ":" + value); }
www.javapolis.com
What's Wrong With This example?
BGGA loop competes with Java 5 for-each Don't define a construct; just implement Iterable
Only compelling use is multiple loop variables Last example doesn't offer power of for-each
Loop variable can’t be primitive (no auto-unboxing) Would require 81 (!) overloadings to do fix this
www.javapolis.com
Loop syntax tailored to for; awkward for while
public static <throws X> void for myWhile( {=> boolean throws X} cond, {=>void throws X} block)
throws X { while (cond.invoke()) { block.invoke(); }}
for myWhile( { => i < 7 } ) { System.out.println(i++); }
myWhile( { => i < 7 }, { => System.out.println(i++); });
www.javapolis.com
Automatic Resource Management Block (BGGA Spec)
<R, T extends Closeable, throws X>R with(T t, {T=>R throws E} block) throws X { try { return block.invoke(t); } finally { try { t.close(); } catch (IOException ex) {} }}
with (FileReader in : makeReader()) { // Requires nesting with (FileWriter out : makeWriter()) { // code using in and out } }
www.javapolis.com
Library-Defined Control Constructs are a Double-Edged Sword
A great feature of Java is “programmer portability” All Java code looks pretty much alike
We can read and maintain each other's code with a
minimum of effort
Library-Defined Control Constructs foster
dialects, which hinder programmer portability
www.javapolis.com
Performance is an Open Question
for-each BGGA % change
List 1.65 s 1.97 s +19%
Array 0.18 s 1.11 s +519%
Time to iterate over 108 elements(Take these numbers with a huge grain of salt)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)Windows XP, Intel T2600 @ 2.16 GHz, 2GB RAMBGGA prototype closures-2007-11-30 (I apologize)
www.javapolis.com
Summary - Programmer-Defined Control Constructs
+ They let you define new control constructs
- They don’t have the same syntax, semantics, or performance as built-in control constructs
- It’s not clear that you need the ability; Java already has a rich set of control structures
- Can lead to dialects
- Responsible for much of the complexity in BGGA- Non-local return, break and continue
- Unrestricted access to mutable local variables
www.javapolis.com
Outline
I. Setting the Stage
II. Why Enhance Support for Closures?
III. BGGA Closures
IV. A Lightweight Approach
V. Where Do We Go From here?
www.javapolis.com
A Lightweight Approach
There is a much simpler approach to reducing the verbosity of anonymous classes and manual resource management
Attack the two problems head-on! Concise syntax for anonymous class instance creation Purpose-built construct for automatic resource
management
www.javapolis.com
Concise Instance Creation Expressions (CICE)
This: sort(list, Comparator<String>(String s1, String s2) { return s1.length() - s2.length(); });
Expands to this: sort(list, new Comparator<String>() { public int compare(String s1, String s2) { return s1.length() - s2.length(); } });
I am not in love with this syntax; we can probably do better.
www.javapolis.com
Automatic Resource Management (ARM) Blocks
// One resource - readFirstLineFromFiletry (BufferedReader r = new BufferedReader(new FileReader(path)) { String s = r.readLine();}
// Multiple resources - copyFiletry (InputStream in = new FileInputStream(src); OutputStream out = new FileOutputStream(dest)) { byte[] buf = new byte[1024]; int n; while ((n = in.read(buf)) >= 0) out.write(buf, 0, n);}
www.javapolis.com
More Automatic Resource Management Blocks
Perhaps this: protected (lock) { ... // access the resource protected by this lock }
Should be shorthand for this: lock.lock(); try { // access the resource protected by this lock } finally { lock.unlock(); }
www.javapolis.com
For More Information
Automatic Resource Management Blocks (ARM) http://docs.google.com/View?docid=dffxznxr_1nmsqkz
Concise Instance Creation Expressions (CICE) http://docs.google.com/View?docid=k73_1ggr36h
Both of these docs are a bit sketchy
www.javapolis.com
Outline
I. Setting the Stage
II. Why Enhance Support for Closures?
III. BGGA Closures
IV. Lightweight Closures
V. Where Do We Go From here?
www.javapolis.com
The Big Questions
Do we really want function types in Java? Do we really want library-defined control
constructs in Java? We should keep in mind that there’s already
a fine programming language for the JVM that offers both of these things, and Java interoperability to boot: Scala
www.javapolis.com
The Medium-Sized Questions
Should we support non-local return, break, and continue? If so: Should it be the default? How do we prevent accidents?
Should we allow access to non-final local variables from closures? If so: Should it be the default? How do we prevent accidents?
www.javapolis.com
Moving Forward, There are Two Approaches
Gain more experience with prototypes When we come to an informed consensus, start a
narrowly focused JSR This might take a couple of years Precedent established by generics
Start a broadly focused JSR in the near future Must permit any point on the Closure Spectrum First task is to answer the two Big Questions
www.javapolis.com
Closing Sermon
We have a big decision to make! It will have a huge effect on the future of the
Java platform We must take our time and do what is right We must not risk further damage to
“the feel of Java”