rubyconf argentina 2011

Who makes the best Asado?

What country has the best footballers?

Argentina!

river

Nacional B???

river

NOOOOOOOO!!!!

Aaron Patterson

@tenderlove

Ruby core team

Rails core team

! WARNING !

ZOMG!

We are in Argentina!

YoJosé

WWFMD?

me gusta

¿Por qué Maria?

THANKS!!

ResourceManagement

Path Management

PatternMatching

Move between theory and practice

Graphvizhttp://graphviz.org

worldhello greeting

digraph nfa { rankdir=LR;

world [shape = doublecircle]; hello [shape = circle]; hello -> world [label="greeting"];}

graph.dot

digraph nfa { rankdir=LR;

world [shape = doublecircle]; hello [shape = circle]; goodbye [shape = circle];

hello -> world [label="greeting"]; goodbye -> world}

graph.dot

world

hellogreeting

goodbye

2 3.

4

/

/articles(.:format)

5

6 /articles/new(.:format)

0 1/ articles (?-mix:[^./?]+)

new

Journey

JourneyYes, it's named after the '70s rock sensation

What is a router?

Provides1: URL generation2: Path Recognition3: Route parsing

Why a new router?

Maintenance

0

750

1500

2250

3000

Journey Journey-2 Rack-Mount

LOC

Known algorithms

References

•Compilers: Principles, Techniques, & Tools (Aho, Lam, Sethi, Ullman)

• Intro to Formal Languages and Automata (Linz)

Predictability

CPU Time

Memory Usage

Current Performance

url generation path recognition route parsing

rack-mount Journey

me gusta

Patterns

Why patterns matter

Regular Expressions/om(g)!/

Rails Routes

resource :articles

/articles(.:format) /articles/new(.:format) /articles/:id/edit(.:format) /articles/:id(.:format)

Parens don't capture

:whatever => /([^./?]+)/

/articles(.:format)

/\/articles(?:\.([^.\/?]+))?$/

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /articles/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /articles/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /articles/new

200 OK!

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

/\/articles(?:.([^.\/?]+))?$/

/\/articles\/new(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/

/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/

GET /foos/new

404 Not Found

How long does this take?

r = routes.length

x = regexp compare

O(r ⨉ x)

Can we do better?

Finite State Machines

Parse Trees

Automata

Parse Trees

Parsing regexp

(a|b)*abb

○○ b

○ b

a *

a b

()

|

Describe node types

Parsing rails routes

Journey::Parserparser = Journey::Parser.newast = parser.parse '/articles(.:format)'

puts ast.to_dot

○

○ ()

/ articles ○

. :format

To String!

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'puts ast.to_sputs ast.to_regexp

OR ASTparser = Journey::Parser.new

trees = [ '/articles(.:format)', '/articles/new(.:format)',].map { |s| parser.parse s }

ast = Journey::Nodes::Or.new trees

puts ast.to_dot

|

○ ○

○ ()

/ articles ○

. :format

○ ()

○ new

○ /

/ articles

○

. :format

SECRET FEATUREparser = Journey::Parser.newast = parser.parse '/articles|books(.:format)'

SECRET FEATUREparser = Journey::Parser.newast = parser.parse '/articles|books(.:format)'

SUPER SECRET!

Automata

(a|b)*abb

3

0b2a

b

1aa

b

b

a

Double circle is acceptance state

baabb

3

0b2a

b

1aa

b

b

a

3

0b2a

b

1aa

b

b

a

b

3

0b2a

b

1aa

b

b

a

ba

3

0b2a

b

1aa

b

b

a

baa

3

0b2a

b

1aa

b

b

a

baabb

aab

3

0b2a

b

1aa

b

b

a

Deterministic Finite Automaton

Only one path

Storageclass DFA attr_reader :table

def initialize @table = Hash.new { |h, from| h[from] = {} } @table[0]['a'] = 1 @table[0]['b'] = 0 @table[1]['a'] = 1 @table[1]['b'] = 2 @table[2]['a'] = 1 @table[2]['b'] = 3 @table[3]['b'] = 0 @table[3]['a'] = 2 endend

FSM Simulation

class Simulator def initialize(dfa) @dfa = dfa end

def simulate(symbols) state = 0 until symbols.empty? state = @dfa.move(state, symbols.shift) end state endend

class DFA ...

def move(from, symbol) @table[from][symbol] endend

move function

irb> sim = Simulator.new(DFA.new)=> #<Simulator:0x007f95929a82e0 ...>irb> sim.simulate %w{ b a a b b }=> 3irb> sim.simulate %w{ a a b }=> 2

Time: O(n)n = string.length

me gusta

Space: S + Tstates.length + transitions.length

Nondeterministic Finite Automaton

Has nil edges

Can't tell direction

Simulation of NFA

602ε

4ε

3a

5b

εε

a|b

602ε

4ε

3a

5b

εε

nil-closure

602ε

4ε

3a ε

5b ε

a

Storageclass NFA def initialize @table = Hash.new { |h, from| h[from] = Hash.new { |i,sym| i[sym] = [] } }

@table[0][nil] << 2 @table[0][nil] << 4 @table[2]['a'] << 3 @table[4]['b'] << 5 @table[3][nil] << 6 @table[5][nil] << 6 end

def nil_closure(states) states.map { |s| @table[s][nil] }.flatten endend

FSM Simulationclass Simulator def initialize(nfa) @nfa = nfa end

def simulate(symbols) states = @nfa.nil_closure([0])

until symbols.empty? next_s = @nfa.move(states, symbols.shift) states = @nfa.nil_closure(next_s) end

states endend

Move function

class NFA ...

def move(states, symbol) states.map { |s| @table[s][symbol] }.flatten endend

irb> sim = Simulator.new(NFA.new)=> #<Simulator:0x007faa188a5f88 ...>irb> sim.simulate %w{ a }=> [6]irb> sim.simulate %w{ b }=> [6]irb> sim.simulate %w{ b b }=> []irb> sim.simulate %w{ c }=> []

Time: O(r ⨉ x)r = operators.length, x = string.length

Who cares about NFA?

NFA Construction

○

○ ()

/ articles ○

. :format

/articles(.:format)

cat nodes

10 /

/articles

20 1/ articles

Optional

30ε

1ε

2????? ε

60 1/ 2articlesε

3ε

4. 5:format ε

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'nfa = Journey::NFA::Builder.new ast

tt = nfa.transition_tableputs tt.to_dot

Journey::NFA

ConvertingNFA to DFA

Eliminate nil transitions

Collapse duplicate edges

60 1/ 2articlesε

3ε

4. 5:format ε

/articles(.:format)

2 3. 40 1/ articles :format

/articles(.:format)

CODES!

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'nfa = Journey::NFA::Builder.new ast

tt = nfa.transition_table.generalized_tableputs tt.to_dot

SHORTER CODES!

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast

tt = dfa.transition_tableputs tt.to_dot

ANY NFAconverts to DFA

O(r ⨉ x) => O(x)r = operations.length, x = string.length

me gusta

Converting Automata to Regexp

2 3. 40 1/ articles :format

/articles(.:format)

2 3. 40 /articles :format

/articles(.:format)

/articles(.:format)

2 4.:format0 /articles

/articles(.:format)

40 /articles(.:format)?

Generalized Transition Graph

/users(.:format) /users/new(.:format) /users/:id/edit(.:format) /users/:id(.:format)

resource :users

2

3.

4/

5

6 8.

7 9/

10

.

11

12 14.

13

15

0 1/ users

(?-mix:[^./?]+)

new(?-mix:[^./?]+)

(?-mix:[^./?]+)

edit

(?-mix:[^./?]+)

(?-mix:[^./?]+)

resource :users

The Plan?

Combine All Routes

Produce DFA

Simulate in O(n) Time

Routing To The Future

JS Simulation

Table => JSON

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast

tt = dfa.transition_tableputs tt.to_json

Table => SVG

parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast

tt = dfa.transition_tableputs tt.to_svg

JS Tokenizer

function tokenize(input, callback) { while(input.length > 0) { callback(input.match(/^[\/\.\?]|[^\/\.\?]+/)[0]); input = input.replace(/^[\/\.\?]|[^\/\.\?]+/, ''); }}

JS Simulator tokenize(input, function(token) { var new_states = []; for(var key in states) { var state = states[key];

if(string_states[state] && string_states[state][token]) { var new_state = string_states[state][token]; highlight_edge(state, new_state); highlight_state(new_state); new_states.push(new_state); } }

if(new_states.length == 0) { return; } states = new_states; });

d3.js

Rails Console

irb> File.open('out.html', 'wb') { |f|irb* f.write(irb* Wot::Application.routes.router.visualizerirb> )}=> 69074

routes.rb marshalling

Test suggestions

Test coverage

Usage Heat Maps

REGEXP

AST

NFA

DFA

ROFLSCALE!!!

DFA => YACC

DFA => RACC

DFA => Ragel

Open Questions

Is our GTG deterministic?

/users/new|/users/:id

4

5

0 1/ 2users 3/new

(?-mix:[^./?]+)

"new" =~ /new|[^.\/?]+/

Can we make it deterministic?

L1 = {new}L2 = {[^./?]+}

/users/new|/users/:id

4

5

0 1/ 2users 3/L1

L2 - L1

Is it worth our effort?

REGEXP

AST

NFA

DFA

Is it worth our effort?

Thank You!!

<3<3<3<3<3