building a python ide with xtext
TRANSCRIPT
B U I L D I N G A P Y T H O N I D E W I T H X T E X T
S E B A S T I A N Z A R N E K O W
T H I S I S N O T A B O U T
T H I S I S A B O U T
A N D A N T L R
A N D
[\n\r\t ]
I N D E N TAT I O N A W A R E
Block: OPEN stmt+=Statement+ CLOSE;
C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S
PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E
C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S
PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S
PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S
Stateless, Upfront Lexing Reentrant Lexing CommonToken Oddities Token Types Only Basic Logic in Terminal Rules
S O M E P Y T H O N E S Q U E C O D E
if name==“Sebastian”: println “Hello”
S O M E P Y T H O N E S Q U E C O D E
if.name==“Sebastian”:\n ....println.“Hello”\n
PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i an ” : \n . . . . p r i n t l n . “ H e l l o ” \n
PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i a n ” : \n . . . . p r i n t l n . “ H e l l o ” \n
CharStream
o.a.IntStream
AntlrStringStream
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
AntlrStringStream
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
TokenSource
Lexer
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
TokenStream
- 1
TokenSource
Lexer
o.a.IntStream
if.name==“Sebastian”:\n....println.“Hello”\n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
9
9
9
9
- 1
if.name==“Sebastian”:\n....println.“Hello”\n
if.name==“Sebastian”:\n....println.“Hello”\n
9
9
9
9
- 1
TokenSource
TokenStream
?
PA R S I N G - T O K E N I Z I N G
Parser
R E C A P
Block: OPEN stmt+=Statement+ CLOSE;
IfStatement: ‘if’ cond=Condition ‘:’ then=Block;
R E C A P
if name==“Sebastian”: println “Hello”
if.name==“Sebastian”:\n....println.“Hello”\n
\n OPEN . . . .
9
9
9
9
- 1
1 3 9
\n CLOSE1 4
PA R S I N G - T O K E N S P L I T T I N G
if.name==“Sebastian”:\n....println.“Hello”\n
9
9
9
9
- 1
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
if.name==“Sebastian”:\n....println.“Hello”\n
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
if.name==“Sebastian”:\n....println.“Hello”\n
PA R S I N G - T O K E N S P L I T T I N G
1 3 9
1 4
TokenStream
Parser
SplittingTokenSource
TokenSource1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Lexer
L E T ’ S D O T H I SI N D E N TA T I O N A W A R E L A N G U A G E S
Block: OPEN stmt+=Statement+ CLOSE;
1. D E F I N E I N D E N TAT I O N T O K E N S
1. D E F I N E I N D E N TAT I O N T O K E N S
terminal OPEN: ‘synthetic:OPEN’; terminal CLOSE: ‘synthetic:CLOSE’;
fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {} .. fragment = parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {}
2. A D J U S T M W E 2 W O R K F L O W
1. GENERATE SPLITTING TOKEN SOURCE 2. DISABLE PARTIAL PARSING
public class MyDslTokenSource extends AbstractIndentationTokenSource { public MyDslTokenSource(TokenSource delegate) { super(delegate); } @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } }
3. M A R K T O K E N S A S S P L I T TA B L E
3. M A R K T O K E N S A S S P L I T TA B L E
protected Token createEndToken(int offset) { CommonToken result = new CommonToken(getEndTokenType()); result.setText(""); result.setChannel(Token.DEFAULT_CHANNEL); result.setStartIndex(offset); result.setStopIndex(offset-1); return result; }
public class AbstractIndentationTokenSource extends AbstractSplittingTokenSource { …
… }
@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }
3. M A R K T O K E N S A S S P L I T TA B L E
@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }
3. M A R K T O K E N S A S S P L I T TA B L E
@Override protected boolean shouldEmitPendingEndTokens() { return false; }
4. M A R K T O K E N S A S S P L I T TA B L E A G A I N1. Configure Content-Assist Parser 2. Almost the same, but:
CharStream
o.a.IntStream TokenSource
Lexer
AntlrStringStream
P U T T I N G E V E R T H I N G T O G E T H E RI N D E N TA T I O N A W A R E L A N G U A G E S
TokenStream Parser
AbstractSplittingTokenSource
AbstractIndentationTokenSource
O N E M O R E T H I N GI N D E N TA T I O N A W A R E L A N G U A G E S
def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }
def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }
5. F O R M AT T I N G
1. Indent Blocks 2. Adjust NewLines 3. Remove superfluous spaces
PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E
D O N EI N D E N TA T I O N A W A R E L A N G U A G E S
Q & A