DIVING DEEPINTO TWIG

or

TWIG INTERNALS

MATTHIAS NOBACKZeist, the NetherlandsFeature addition in March 2014Started as a web designer (2003)Was employed for 6 years atseveral companies...

Now self employed: Blog: Twitter:

Noback's Officephp-and-symfony.matthiasnoback.nl

@matthiasnoback

leanpub.com/a-year-with-symfony

@matthiasnoback enjoyed reading your book ­ should be mandatory reading for any Symfony developer.6:30 PM ­ 7 Sep 2013

Tony Piper @tonypiper

Follow

@matthiasnoback An excellent job, good sir. Your blog posts and now your book are really showing us the way to be better developers.10:47 PM ­ 5 Sep 2013

Damon Jones @damon__jones

Follow

ARMIN RONACHERDeveloped Jinja for Python (2006)Ported Jinja to PHP, called it Twig(2008)

FABIEN POTENCIERLead developer of the Symfony projectWas looking for a Django-like templatingengine for Symfony2Found Twig and started "hacking" on it

OVERVIEWSampleExtensionsLexerParserToken parsersNode visitorsCompiler

SAMPLES

BLOCKS AND VARIABLES{% for user in users %} <li>{{ user.name }}</li>{% endfor %}

FUNCTIONSI am {{ random(['happy', 'sad']) }}

{% set steps=range(1, 20) %}

<li class="{{ cycle(['odd', 'even'], i) }}">...</li>

FILTERSHi, {{ user.name|capitalize }}!

{{ user.phoneNumbers|first }}

{{ post.tags|join(', ') }}

TESTS{% if expiryDate is defined %}...

{% if users is empty %}...

{% if i is odd %}

TAGS{% if user.enabled %}...

{% block sidebar %}...

{% include '_profile.html.twig' %}...

EXTENDINGTWIG

(Symfony2: create a service with a tag)

EXTENSIONSinterface Twig_ExtensionInterface{ public function getFunctions();

public function getFilters();

public function getTests();

...}

$env->addExtension($extension) // is the way

twig.extension

FUNCTIONSclass MyExtension extends \Twig_Extension{ public function getFunctions() { return array( new \Twig_SimpleFunction( 'myFunction', function ($thing) { return sprintf('This is <b>my</b> %s.', $thing); } ) ); }}

{{ myFunction("computer") }}

FILTERSclass MyExtension extends \Twig_Extension{ public function getFilters() { return array( new \Twig_SimpleFilter( 'mine', function ($what, $mine = true) { return sprintf( '%s (which %s mine)', $what, $mine ? 'is':'is not' ); } ) ); }}{{ thing|mine(false) }}

TESTSclass MyExtension extends \Twig_Extension{ public function getTests() { return array( new \Twig_SimpleTest( 'a_conference', function($name) { return $name === 'SymfonyCon'; } ) ); }}

{% if "SymfonyCon" is a_conference %}I told you so{% endif %}

TAGS{% conference %}

Needs some explaining...

LOADING ATEMPLATE

TWIG ENVIRONMENT$env = new \Twig_Environment();

$env->setLoader(new \Twig_Loader_Filesystem(__DIR__.'/templates'));

$context = array( ...);

echo $env->render('index.html.twig', $context);

WHAT IS A TEMPLATE?class Twig_Environment{ ...

public function render($name, array $context = array()) { return $this->loadTemplate($name)->render($context); }}

A template is a class that implements\Twig_TemplateInterfaceloadTemplate() returns an instance of such a class

COMPILED TEMPLATE CLASS/* index.html.twig */class __TwigTemplate_d1d2705938bfae31fd9839ce0fe15e96 extends Twig_Template{ protected function doDisplay(array $context, array $blocks = array()) { // line 1 if (isset($context["name"])) { $_name_ = $context["name"]; } else { $_name_ = null echo twig_escape_filter($this->env, $_name_, "html", null, true); echo " is in a file"; }

public function getTemplateName() { return "index.html.twig"; }}

BEFORE<p>{{ name }}</p>

AFTERecho '<p>';

if (isset($context["name"])) { $_name_ = $context["name"];} else { $_name_ = null;}

echo twig_escape_filter($this->env, $_name_, "html", null, true);

echo '</p>';

HOW TWIG CREATES ATEMPLATE CLASS

1. Retrieve the source (written in "Twig") from the loader(s)2. Compile the source to a PHP class

COMPILING A TEMPLATE

THE LEXER

LEXER1. Matches the input string against known patterns

("lexemes")2. Determines token types for these matches3. Returns a stream of tokens

THE LEXER IN YOUR MIND

FINDING TWIG BLOCKSThe lexer first checks for the position of the main markers:

Start of block: {%Start of variable: {{Start of comment: {#

Then the lexer

1. iterates over the resulting positions, while2. checking some basic syntax rules, and3. collecting tokens on its way to EOF

TOKEN TYPESTokens have:

a typea value (optional)a line number

BLOCK_START {%BLOCK_END %}VAR_START {{VAR_END }}TEXT raw template dataNAME for, if, etc.NUMBER a numberSTRING "..." or '...'OPERATOR +, *, ~, etc.PUNCTUATION |, [, {, etc.... ...

Take this template:{% endif %}<ul> {% for item in items %} <li>{{ item|capitalize }}</li> {% endfor %}</ul>

$lexer = $env->getLexer();

$template = ...;

$tokenStream = $lexer->tokenize($template);

TOKEN STREAMecho $tokenStream;

BLOCK_START {%NAME(endif) endifBLOCK_END %}TEXT(<ul>) raw template

dataBLOCK_START {%NAME(for) forNAME(item) itemOPERATOR(in) inNAME(items) itemsBLOCK_END %}... ...EOF end of input

{% endif %}<ul> {% for item in items %} <li>{{ item|capitalize }}</li> {% endfor %}</ul>

STATESTo keep track of what the lexer is doing.

DATA lexing raw template data (start state)

BLOCK lexing a block

VAR lexing a variable

STRING lexing a string

CONSECUTIVE STATESDATA template dataBLOCK block endif

startsBLOCK block endif

endsDATA <ul>BLOCK block for startsBLOCK name: itemBLOCK name: inBLOCK name: itemsBLOCK block for endsDATA <li>VAR variable starts,

name: item... ...DATA </ul>

{% endif %}<ul> {% for item in items %} <li>{{ item|capitalize }}</li> {% endfor %}</ul>

SYNTAX VALIDATIONEach block and variable should be closed

Brackets ({[ should be closed symmetrically

Closing brackets ]}) can not occur first

{% for {% if

{{ ['a' }}

{{ ] }}

SYNTAX VALIDATION(CONTINUED)

Expressions may not contain unexpected characters

Comments should be closed

{{ \ }}

{# comment

FROM SYNTAX TOSEMANTICS

The resulting list of tokens may be semantically incorrect.

In the Twig language, that is...{% endif %}<ul> {% for item in items %} <li>{{ item|capitalize }}</li> {% endfor %}</ul>

COMPILING A TEMPLATE

THE PARSER

PARSING THE TOKENSTREAM

The parser

Processes the token streamBuilds an Abstract Syntax Tree for the template

CREATING THE ABSTRACTSYNTAX TREE

<ul> {% for item in items %} <li>{{ item|capitalize }}</li> {% endfor %}</ul>

$template = ...;

$tokenStream = $lexer->tokenize($template);

$parser = $env->getParser();

$nodeTree = $parser->parse($tokenStream);

echo $nodeTree;

EXCERPT OF THE ABSTRACTSYNTAX TREE

Twig_Node_Module( body: Twig_Node_Body( 0: Twig_Node( 0: Twig_Node_Text(data: '<ul>') 1: Twig_Node( 0: Twig_Node_SetTemp( name: 'items' ) 1: Twig_Node_For( value_target: Twig_Node_Expression_AssignName( name: 'item' ) seq: Twig_Node_Expression_TempName( name: 'items' ) body: Twig_Node( 0: Twig_Node( 0: Twig_Node_Text( data: '<li>' ) 1: Twig_Node( 0: Twig_Node_SetTemp( name: 'item' ) 1: Twig_Node_Print( expr: Twig_Node_Expression_Filter( node: Twig_Node_Expression_Filter(

THE ROOT NODEThe parsing process results in a root node containing

the body of the template,or a collection of blocks,the link to a parent template,...

THE MAIN TOKEN TYPESThe parser collects nodes based on the token at the currentposition in the token stream.

TEXT template text create a Text node with the value of the

token

VAR_START variable parse the expression that follows and

expect VAR_ENDBLOCK_STARTblock with a tag expect a name, which is the name of the

tag (i.e. for, if, etc.) and call a subparser

SUBPARSER == TOKENPARSER

Each token parser defines its own rules for the tokens thatshould follow the tag:

{% for item in items %}

{% include 'template.html' with {'foo': 'bar'} %}

{% set foo, bar = 'foo', 'bar' %}

FROM TOKENS TO NODESThe token parser returns nodes based on the tokens it finds.

Returned nodes are inserted in the Abstract Syntax Tree

A CUSTOM TOKEN PARSERclass ConferenceTokenParser extends \Twig_TokenParser{ public function parse(\Twig_Token $token) { $this->parser->getStream()->expect(\Twig_Token::BLOCK_END);

$expr = new \Twig_Node_Expression_Constant('SymfonyCon', $token->getLine());

return new \Twig_Node_Print($expr, $token->getLine(), $this->getTag()); }

public function getTag() { return 'conference'; }}

{% conference %}

There we go, {% conference %}!

$template = ...;

$env = new \Twig_Environment();

$env->addTokenParser(new ConferenceTokenParser());

echo $env->parse($env->tokenize($template));

(Better: register them using the getTokenParsers() ofyour Twig extension class)

EXCERPT OF THE ABSTRACTSYNTAX TREE

Twig_Node_Module( body: Twig_Node_Body( 0: Twig_Node( 0: Twig_Node_Text( data: 'There we go, ' ) 1: Twig_Node_Print( expr: Twig_Node_Expression_Constant( value: 'SymfonyCon' ) ) 2: Twig_Node_Text( data: '!' ) ) ))

EXPRESSIONS{{ 5 + age * 4 }}

$template = ...;

$moduleNode = $env->parse($env->tokenize($template));

echo $moduleNode;

Expressions are parsed by a specialized expression parser.

EXPRESSIONS...expr: Twig_Node_Expression_Binary_Add( left: Twig_Node_Expression_Constant(value: 5) right: Twig_Node_Expression_Binary_Mul( left: Twig_Node_Expression_TempName(name: 'age') right: Twig_Node_Expression_Constant(value: 4) ))

ASSOCIATIVITYMost operators are left associative,which means that

a + b + c

is to be read as((a + b) + c)

and not as(a + (b + c))

PRECEDENCEOperators have a number indicatingtheir precedence, so

a + b * c

will always be interpreted as(a + (b * c))

instead of(a + b) * c)

ALSO: NODE VISITORSAllowed to revisit the entire node tree and change anything.

E.g. auto-escaping

COMPILING A TEMPLATE

THE COMPILER

THE ROOT NODEThe node tree contains nodes generated by:

the parserthe expression parsertoken parsersnode visitors

THE COMPILE STEP{% for item in items %} {{ item|capitalize }}<br>{% endfor %}

$template = ...;

echo $env->compileSource($template);

EXCERPT OF THE COMPILEDTEMPLATE

class __TwigTemplate_d41d8cd98f00b204e9800998ecf8427e extends Twig_Template{ protected function doDisplay(array $context, array $blocks = array()) { // line 1 if (isset($context["items"])) { $_items_ = $context["items"]; } else { $_items_ = $context['_parent'] = (array) $context; $context['_seq'] = twig_ensure_traversable($_items_); foreach ($context['_seq'] as $context["_key"] => $context["item"]) { // line 2 echo " "; if (isset($context["item"])) { $_item_ = $context["item"]; } else { $_item_ = echo twig_escape_filter($this->env, twig_capitalize_string_filter($this->env, echo "<br>"; } $_parent = $context['_parent']; unset($context['_seq'], $context['_iterated'], $context['_key'], $context['item'], $context = array_merge($_parent, array_intersect_key($context, $_parent)); }}

RECURSIVE COMPILINGThe compiler just calls the compile()method of the root node.

Which calls the compile() of childnodes, etc, etc.

CUSTOM TOKEN PARSERREVISITED

class ConferenceTokenParser extends \Twig_TokenParser{ public function parse(Twig_Token $token) { ...

$expr = new \Twig_Node_Expression_Constant('SymfonyCon', $token->getLine());

return new \Twig_Node_Print($expr, $token->getLine(), $this->getTag()); }}

COMPILING A PRINT NODEclass Twig_Node_Print{ public function compile(Twig_Compiler $compiler) { $compiler ->addDebugInfo($this) ->write('echo ') ->subcompile($this->getNode('expr')) ->raw(";\n") ; }}

The result can be found in the compiled template:protected function doDisplay(array $context, array $blocks = array()){ // line 1 ... echo "SymfonyCon"; ...}

SOME REFLECTIONS1. You can put any PHP code you want inside a template2. You can do (heavy) calculations at compile time (just once)

You only have to create your own node type andimplement its compile method.

TESTING AND DEBUGGINGWriting parsers and nodes can be quite difficult, so

1. read the compiled templates in your cache directory and2. write unit tests for your custom parser and node type

(extend from \Twig_Test_NodeTestCase)

THAT'S ALL

QUESTIONS?

THANK YOUAND GOOD BYE

joind.in/10368

nobacksoffice.nl

php-and-symfony.matthiasnoback.nl

@matthiasnoback

REFERENCESArmin's blogJinja documentationTemplating engines in PHPTemplating engines in PHP - Follow-upArticle about node visitors

IMAGEShttp://www.stockfreeimages.com/http://twig.sensiolabs.org/