goto java; (jfokus)

@MSkarsaune

Martin SkarsauneJava Developer and Co-Owner

A peek into the OpenJDK compiler : goto java;

高馬丁

We’re

Hiring!

@MSkarsaune

GOTO Statements in Java!

@MSkarsaune

GOTO Statement – Objective• Syntax

goto identifier;• Runtime• Program control moves to target statement

• Other Semantics • Target (statement label) must be defined in same

scope, compilation error if not• Potential circular gotos should give compilation

warning.

@MSkarsaune

GOTO Statement – Means• OpenJDK• Open source Java implementation• Javac is implemented in plain Java

• Modify compiler to support GOTO

@MSkarsaune

public class GotoSuccess {

public static void main(String[] args) {one: System.out.print("goto ");two: System.out.print(”J");goto four;three: System.out.print(”2017");goto five;four: System.out.print(”Fokus ");goto three;five: System.out.print("!");

Success Case

@MSkarsaune

GOTOAHEAD

Dijkstra 1968:

@MSkarsaune

What is a Compiler?

Compiler

@MSkarsaune

What is a Compiler?

frontend

backend

@MSkarsaune

Parse Enter Process Attribute Flow Desugar Generat

processAnnotations(enterTrees(… parseFiles(…))),…)

…generate(desugar(flow(attribute(…))))

@MSkarsaune

• Syntax: goto identifier;• First convert character stream to token stream

(Scanner.java)

GOTO IDENTIFIER SEMI

[g][o][t][o][ ][f][o][u][r][;]

@MSkarsaune

• goto is already a reserved word in Java!• Lucky for us, goto is therefore defined as a

TokenKind. • Tokens.java:141: GOTO(“goto”)• The scanner therefore works out of the box!

@MSkarsaune

...import com.sun.tools.javac.parser.Scanner;...

final Scanner scanner=factory.newScanner(”goto identifier;”, false);

@MSkarsaune

scanner.nextToken();

@MSkarsaune

scanner.nextToken();assertThat(”First token is GOTO”, scanner.token().kind, is(GOTO));

@MSkarsaune

scanner.nextToken();assertThat("Second token is IDENTIFIER", scanner.token().kind,

is(IDENTIFIER));

@MSkarsaune

is(IDENTIFIER));

@MSkarsaune

is(IDENTIFIER));

scanner.nextToken(); assertThat("Third token is SEMI", scanner.token().kind, is(SEMI));

@MSkarsaune

CompilationUnit

ClassDefinitionFieldDefintion

MethodDefinition

Signature

BodyIfStatement

visitClassDef(..)

visitMethodDef(..)

visitIf(..)

Abstract Syntax Tree

[g][o][t][o][ ][f][o][u][r][;]

GOTO IDENTIFIER SEMIWikipedia: “the visitor design pattern is a way of separating an algorithm from an

object structure on which it operates”

@MSkarsaune

ClassInterface

@MSkarsaune

Interface based visitors

@MSkarsaune

Class based visitors

public void visitGoto(JCGoto tree) { try { print("goto " + tree.label + ";"); } catch (IOException e) { throw new UncheckedIOException(e); } }

public void visitGoto(JCGoto tree) { //TODO implement}

@MSkarsaune

public static class JCLabeledStatement extends JCStatement implements LabeledStatementTree {…public GotoResolver handler;…}

public class GotoResolver { Map<GotoTree, Name> gotos; Map<Name, LabeledStatementTree> targets; List<StatementTree> statementsInSequence; ...}

Helper object

@MSkarsaune

JavacParser.parseStatement()

case GOTO: { nextToken(); Name label = ident(); JCGoto t = to(F.at(pos).Goto(label, getGotoResolver()); accept(SEMI); return t; }

TreeMaker.java:

public JCGoto Goto(Name label, GotoResolver resolver) { JCGoto tree = new JCGoto(label, resolver); tree.pos = pos; return tree;}

@MSkarsaune

…generate(desugar(flow(attribute(…))))✔

@MSkarsaune

• Basic sanity testing of compilation unit:

• File name and folder location• Duplicate class names

• Corrections• Add default constructor if no

constructors are declared

@MSkarsaune

Default Constructorpublic class SimpleClass {

@MSkarsaune

Default Constructorpublic class SimpleClass { public SimpleClass() { super(); }}

@MSkarsaune

✔Parse Enter Process Attribut

e Flow Desugar Generate

@MSkarsaune

• Process Annitations• Annotation processing API• Part of ordinary javac process since Java

1.6• Plugin API (see javac documentation)

@MSkarsaune

• Output controlled by command line switches

- proc:only – only process annotations, do not compile- proc:none – do not process annotations

@MSkarsaune

• Attribution• Semantic checks

–Types–References

• Corrections–Add required calls to super constructor

@MSkarsaune

Ensure target label exists in current scopepublic class GotoMissingLabel {

public static void main(String[] args) {one: System.out.print("goto ");two: System.out.print(”Java");goto six;three: System.out.print(”2016");goto five;four: System.out.print(”One ");goto three;five: System.out.print("!");

@MSkarsaune

Attr.java: @Override public void visitGoto(JCGoto that) {

that.findTarget();if(that.target==null)

log.error(that.pos(), "undef.label", that.label);result = null;

class JCGoto: …public void findTarget() { this.target = (JCLabeledStatement)this.handler.findTarget(this);}

@MSkarsaune

Flow analysis• Detect unreachable code• Verify assignments• Ensure proper method return• Verify (effectively) final• Check exception flow

@MSkarsaune

Detect circular gotospublic class GotoCircularWarning {

public static void main(String[] args) { one: System.out.print("goto "); two: System.out.print(”Java"); goto four; three: System.out.print(”2016"); goto five; four: System.out.print(”One "); goto three; five: System.out.println("!"); goto one;//forms infinite loop }}

@MSkarsaune

Flow.FlowAnalyzer class:

compiler.properties:

@Override public void visitGoto(JCGoto tree) { if (tree.handler.detectCircularGotoPosition(tree))

log.warning(tree.pos, "circular.goto"); }

...compiler.warn.circular.goto=circular goto...

@MSkarsaune

✔processAnnotations(enterTrees(…

parseFiles(…))),…)…

generate(desugar(flow(attribute(…))))

@MSkarsaune

Erase generic types

public class Bridge implements Comparator {

TransTypes Unlambda Lowerpublic interface Comparator<T> {int compare(T o1, T o1);

}or<T> {

@MSkarsaune

Erase generic types

public class Bridge implements Comparator<Integer> {

public int compare(Integer first, Integer second) {return first - second;

TransTypes Unlambda Lower

public interface Comparator<T> {int compare(T o1, T o1);

} or<T> {

@MSkarsaune

Erasure - Runtime

@MSkarsaune

Erasure - Bridge

} /*synthetic*/ public int compare(Object first, Object second) { return this.compare((Integer)first, (Integer)second); }

@MSkarsaune

TransTypes Unlambda LowerExtract inner classpublic class Outer {

private void foo() { }

public Runnable fooRunner() {return new Runnable() {

public void run() {foo();

@MSkarsaune

Extract inner class

public class Outer {private void foo() { }

public Runnable fooRunner() {return new Runnable() {

public void run() {foo();

@MSkarsaune

Extract inner classpublic class Outer {

public Runnable fooRunner() {return new Outer$1(this);

class Outer$1 implements Runnable {final Outer this$0;

Outer$1(final Outer this$0) { this.this$0 = this$0; super(); } public void run() {

this$0.foo(); }}

@MSkarsaune

Extract inner class

public class Outer {private void foo() { }

this$0.foo(); }}

@MSkarsaune

Extract inner classpublic class Outer {

}/*synthetic*/static void access$000(

Outer x0) { x0.foo(); }

Outer.access$000(this$0); }}

@MSkarsaune

TransTypes Unlambda LowerBoxingList<Integer> list =

Arrays.asList(1, 2);for (Integer i : list) {

System.out.println(”Double: “ + i * 2);}

@MSkarsaune

Boxing

List<Integer> list = Arrays.asList(1, 2);

for (Integer i : list) {System.out.println(”Double: “ + i * 2);

public static <T> List<T> asList(T... a) { return new ArrayList<>(a); }

@MSkarsaune

TransTypes Unlambda LowerBoxingList<Integer> list =

Arrays.asList(Integer.valueOf(1), Integer.valueOf(2));for (Integer i : list) {

@MSkarsaune

TransTypes Unlambda LowerUnboxingList<Integer> list =

@MSkarsaune

TransTypes Unlambda LowerUnboxingList<Integer> list =

System.out.println(”Double: “ + i.intValue() * 2);}

@MSkarsaune

TransTypes Unlambda LowerVarargsList<Integer> list =

public static <T> List<T> asList(T... a) { return new ArrayList<>(a); }

@MSkarsaune

TransTypes Unlambda LowerVarargsList<Integer> list =

public static <T> List<T> asList(T... a) { return new ArrayList<>(a);}

@MSkarsaune

TransTypes Unlambda LowerVarargs - runtimeList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Integer i : list) {

@MSkarsaune

TransTypes Unlambda LowerFor each loopList<Integer> list =

@MSkarsaune

public interface Iterable<T> { Iterator<T> iterator();}

@MSkarsaune

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (;;) {

@MSkarsaune

For each loopList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator();;) {

Integer i System.out.println(”Double: “ + i.intValue() * 2);

@MSkarsaune

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator(); i$.hasNext();) {

Integer i System.out.println(”Double: “ + i.intValue() * 2);

@MSkarsaune

For each loopList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator(); i$.hasNext();) {

Integer i = (Integer)i$.next(); System.out.println(”Double: “ + i.intValue() * 2);

@MSkarsaune

TransTypes Unlambda LowerEnumspublic enum Status {

YES, NO, MAYBE}

@MSkarsaune

Enums - constructorpublic enum Status {

YES, NO, MAYBEprivate Status(String $enum$name, int $enum$ordinal) {

super($enum$name, $enum$ordinal); }}

public static final Status TRUE = new Status("TRUE", 0);public static final Status FALSE = new Status("FALSE", 1);public static final Status MAYBE = new Status("MAYBE", 2);

@MSkarsaune

TransTypes Unlambda LowerEnums - valueOfpublic enum Status {

super($enum$name, $enum$ordinal); } public static Status valueOf(String name) { return (Status)Enum.valueOf(Status.class, name); }}

@MSkarsaune

TransTypes Unlambda LowerEnums - valuespublic enum Status {

super($enum$name, $enum$ordinal); } public static Status valueOf(String name) { return (Status)Enum.valueOf(Status.class, name); } private static final Status[] $VALUES = new Status[]{ Status.YES, Status.NO, Status.MAYBE};

public static Status[] values() { return (Status[])$VALUES.clone(); }}

@MSkarsaune

TransTypes Unlambda LowerEnum switch statementpublic class SwitchStatus {

void switchStatus(Status status) {

switch (status) { case MAYBE: return;

default: break; } }}

@MSkarsaune

switch (status) { case MAYBE: return;

@MSkarsaune

Enum switch statementpublic class SwitchStatus {

switch (SwitchStatus$1.$SwitchMap$Status[(status).ordinal()]) { case 1: return;

class SwitchStatus$1 {

@MSkarsaune

Enum switch statementpublic class SwitchStatus {

class SwitchStatus$1 {static final int[] $SwitchMap$Status = new

int[Status.values().length];

@MSkarsaune

int[Status.values().length]; [0][0][0]

@MSkarsaune

int[Status.values().length]; [0][0][1]static { try { SwitchStatus$1.$SwitchMap$Status[Status.MAYBE.ordinal()] = 1; } catch (NoSuchFieldError ex) { } }}

@MSkarsaune

• Organize initializers• String concatenation• Generate bytecodes

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key="key"; String value; public InstanceInitialization(String value) { this.value = value; }

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key="key"; String value; public InstanceInitialization(String value) {

super(); this.value = value; }

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key; String value; public InstanceInitialization(String value) {

super(); key = ”key”; this.value = value; }

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key; String value; public void <init> () {

super(); key = ”key”; this.value = value; }

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key="key"; static { init(); }}

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key; static { key="key"; init(); }}

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key; static void <clinit>() { key="key"; init(); }}

@MSkarsaune

String concatenationSource code “Generated code”

”string” + value

@MSkarsaune

”string” + value new StringBuilder()

@MSkarsaune

”string” + value new StringBuilder() .append(”string”)

@MSkarsaune

”string” + value new StringBuilder() .append(”string”) .append(value)

@MSkarsaune

”string” + value new StringBuilder() .append(”string”) .append(value) .toString()

@MSkarsaune

• Goto generation–Luckily for us there is a GOTO byte code–goto <addr>

@MSkarsaune

if (<test>) { <ifblock>} else { <elseblock>}<codeafter>

Source code Byte codeCI

9 ifeq<elseblock>

goto<stackmap>

22 <ifblock>

Java >= 1.6:Stack map frames must be embedded at target of jump instruction

Code generator (Code.java) marked as not alive.Goto instruction added to list of pending jumps (Chain.java)

Pending jumps processed

Normal GOTO usage

@MSkarsaune

Source code Byte codeCI…label: <somecode>…goto label;

… goto 20

Used by goto?Must emit stackmap

Emit goto to label and turn code generation on again

GOTO scenario 1 : jump back

@MSkarsaune

Source code Byte codeCI

GOTO scenario 2 : jump forward…goto label; …label: <somecode>

……

29 <somecode>

goto<stackmap>

Label position not yet known?• emit goto• add to list of pending gotos• turn generation on again

29Label used? • emit stack frame • patch pending

@MSkarsaune

• Goto generation–Gen.java , visitor for code generation–Modify for LabelledStatement–Add implementation for Goto

@MSkarsaune

• Gen.java – Labelled Statement

public void visitLabelled(JCLabeledStatement tree) { // if the label is used from gotos, have to emit stack map if (tree.handler.isUsed(tree)) code.emitStackMap(); … }

@MSkarsaune

• Gen.java – Goto

public void visitGoto(JCGoto tree) { tree.handler.addBranch(new Chain(code.emitJump(goto_), null, code.state.dup()), tree.target); //normally goto marks code as not alive, turn generation on code.entryPoint();}

Target position known?• Yes – patch immediately• No – add to list of pending gotos

@MSkarsaune

ambdaλ

@MSkarsaune

• Lambda implementation in Java 8–Language change–Compilation–Runtime support

• Many interesting design considerations

@MSkarsaune

Simple Examplepublic Comparator<String> lambdaExample() { return (String a, String b) -> a.compareTo(b);}

@MSkarsaune

LambdaToMethod.javapublic Comparator<String> lambdaExample() { return (String a, String b) -> a.compareTo(b);}

/*synthetic*/ private static int lambda$lambdaExample$0( , ) { return ;}

final String afinal String b

a.compareTo(b)

@MSkarsaune

Runtimepublic Comparator<String> lambdaExample() { return <invokedynamic>LambdaMetafactory.metafactory(}

/*synthetic*/ private static int lambda$lambdaExample$0( final String a, final String b) { return a.compareTo(b);}

Lookup(LambdaExample), /*caller*/ "compare", ()Comparator, /*MethodType*/ (Object,Object)int, /*MethodType*/ lambda$lambdaexample$0,/*MethodHandle*/ (String,String)int); /*MethodType*/

final class LambdaExample$$Lambda$1/1834188994 implements Comparator { private LambdaExample$$Lambda$1/1834188994() public int compare(Object,Object)}

public interface Comparator {/*erased*/ int compare(Object o1, Object o2);}

@MSkarsaune

Runtime implementation

LambdaMetaFactory• metaFactory(…)• altMetaFactory(…)

InnerClassLambdaMetafactory

Lambda Class

@MSkarsaune

SerializationLambda Instance

Serialize

SerializedLambda

Deserialize

LambdaMeta

Factory

@MSkarsaune

• Possible to back port ?–Capture generated class ?–Compile time generation of inner class!

@MSkarsaune

Step 1: Source.java

public boolean allowLambda() { return compareTo( ) >= 0;}

JDK1_8JDK1_5

@MSkarsaune

Step 2: Special handling

boolean mustBackportLambda() { return this.target.compareTo(Target.JDK1_8) < 0;}

@MSkarsaune

Step 3: Call backport

if(!this.attr.mustBackportLambda()) { result = makeMetafactoryIndyCall(...);}else{ result = new LambdaBackPorter(...).implementLambdaClass(...);}

@MSkarsaune

Example implementationprivate static final class Lambda$$2 implements Comparator<String> {

Lambda$$2 { super(); } public int compare(String arg0, String arg1) { return LambdaExample.lambda$lambdaExample$0(arg0, arg1); } public int compare(Object o1, Object o2) { return this.compare((String)o1, (String)o2); }}

@MSkarsaune

Example invokingpublic Comparator<String> lambdaExample() { }

return LambdaMetafactory.metafactory(...);return new Lambda$$2();

@MSkarsaune

ambdaλ

@MSkarsaune

• <invokedynamic> “playground”1. @InvokeIndy annotation on method with

reference to boostrap method2. Annotation processor that validates

reference3. Compiler plugin that replaces method

invocation with invokedynamic

@MSkarsaune

@IndyMethodpublic class SomeService {

//must refer to a valid public static method //with certain characteristics

@IndyMethod(implementation="no.kantega.jvm.indy.example.SomeProvider", method= "dummyMethod")

public static void doStuff() {throw new UnsupportedOperationException("...");

@MSkarsaune

no.kantega.jvm.indy.compiler.plugin.IndyAnnotationChecker

Hooking in the processor:META-INF/services/javax.annotation.processing.Processor:

@MSkarsaune

Annotation processor: setup, compliance + mapping@SupportedAnnotationTypes("no....IndyMethod")

@SupportedSourceVersion(SourceVersion.RELEASE_8)//Java versionpublic class IndyAnnotationChecker extends AbstractProcessor {

//...public boolean process(Set<...> annotations, RoundEnvironment roundEnv) {

for (TypeElement annotation : annotations) {for (Element element : roundEnv.getElementsAnnotatedWith(annotation)) {

//...raise error if earlier than Java 7 or missing pluginIndyMethod indyMethodRef = element.getAnnotation(IndyMethod.class);if (indyMethodRef != null) {

//...check existance of type and compliance of method ...processingEnv.getMessager().printMessage(Kind.ERROR,"...", element);}

}//...

@MSkarsaune

...<build>

<artifactId>maven-compiler-plugin</artifactId><configuration>

<compilerArgument>-proc:none</compilerArgument>

</configuration></plugin>

</plugins></build>

Disable annotation processing in the project that implements the plugin (!): pom.xml :

@MSkarsaune

• Compiler plugin–Hook straight into the compilation process–Respond to events from compilation process–Make changes to AST

@MSkarsaune

no.kantega.jvm.indy.compiler.plugin.IndyPlugin

Hooking in the plugin:META-INF/services/com.sun.source.util.Plugin:

@MSkarsaune

public class IndyPlugin implements Plugin {public String getName() {

return "IndyPlugin";}public void init(JavacTask paramJavacTask,

String... paramArrayOfString) {paramJavacTask.addTaskListener(

new GenerateInvokeDynamicHandler());}

Typical plugin definition:• Unique name• Delegate to task listener(s)

@MSkarsaune

public class GenerateInvokeDynamicHandler implements TaskListener {

public void started(TaskEvent start) {if(start.getKind() == Kind.GENERATE) {

for (Tree tree : start.getCompilationUnit().getTypeDecls()) {tree.accept(new IndyMethodInvocationReplacer(), tree);

Task listener:• Receive callback, check stage• Insert visitor to process ASTs

@MSkarsaune

public class IndyMethodInvocationReplacer extends TreeScanner {public Object visitMethodInvocation(MethodInvocationTree node, Tree p) {{//...various checks on the method call//...see if annotation processor has created mapping for itMethodSymbol replacementMethod = IndyMethodMappings.getInstance().

mappingFor((MethodSymbol) identifier.sym);if(replacementMethod!= null) {//insert reference to bootstrap

identifier.sym=new Symbol.DynamicMethodSymbol(...); }

}return super.visitMethodInvocation(node, p);

Visitor:• Make modifications

@MSkarsaune

<plugin><artifactId>maven-compiler-plugin</artifactId><configuration>

<compilerArgs><arg>-Xplugin:IndyPlugin</arg>

</compilerArgs><source>1.8</source><target>1.8</target>

</configuration></plugin>

</plugins></build>

Users:• Enable plugin in compilation

@MSkarsaune

Wrap Up• The Java compiler is written in pure Java• Compilation is done in phases• Programming language advances (syntactic sugar)

require good compiler support• Lambdas are compiled in a forward compatible manner• Annotation processors and compiler plugins may be

used to tailor the compilation process to specific needs

@MSkarsaune

Resources• OpenJDK

• Source: http://hg.openjdk.java.net/• Compiler hacking tutorial

• http://www.ahristov.com/tutorial/java-compiler.html• Sample code:

• https://github.com/skarsaune/goto• https://github.com/skarsaune/indy-plugin

• Slide pack:• http://www.slideshare.net/MartinSkarsaune

• 60 minute video (Devoxx) :• https://youtu.be/gGTDQq6ZjIk

@MSkarsaune

Questions or Comments?Martin Skarsaune

Java Developer and Co-Owner

@MSkarsaune

Thank You for Your Time!Martin Skarsaune

Java Developer and Co-Owner

goto java; (jfokus)

Software

embedded java: smart, connected,...

jfokus 2012 : the java ee 7 platform: developing for the...

javazone 2014 - goto java;

jruby + rails - jfokus.pdf · jruby + rails = awesome java...

android jumpstart jfokus

visualizing java code bases for jfokus 2017

down and dirty with java ee 7 - jfokus and dirty with java...

jfokus functional groovy

devoteam group - jfokus

jruby + rails = awesome java web framework at jfokus 2011

java fx and java se 6 update n - jfokus · java fx and java...

jfokus 50 new things with java 8

spring framework 4 on java 8 - goto...

intro to mongodb & the jvm - jfokus · intro to mongodb &...

binary patching of java classes for fun and profit - jfokus...

java language basics by . keywords keywords of java are...

real world java 9 - goto...

introducing scala in java territory - goto...

keeping up with java - goto conference...nov 02, 2018 ·...

kanban kata - jfokus