transform your state \/ err

$: Transform your State \/ Err$
@gerferra

Transform your State \/ Err

Transform your State \/ Err

An example of functional parsing with state and error handling

An example of functional parsing with state and error handling

@gerferra

Initial language

Arithmetic expressions over integers

1;

2+2;

3*3+4*4*(1+2);

Functional parsing

Write a parser using “just functions” ...

… within some programming language

… in the form of some embedded DSL

Functional parsingAll of this has a proper name:

Parser combinators

A collection of basic parsing functions that recognise a piece of input

A collection of combinators that build new parsers out of existing oneshttp://www.fing.edu.uy/inco/cursos/pfa/uploads/Main/parsercombinators

Parser combinatorsScala has many parsing combinator libraries

Two common ones:

scala-parser-combinators

parboiled2


Two common ones:


parboiled2

The one we will be using


Two common ones:


parboiled2 Not really parser combinators ...

but very fast!


Two common ones:


parboiled2

New kid on the block: lihaoyi/fastparse


Part of the Scala standard library but distributed separately

libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.4"


Slow but good enough in this case

The first (?) parser combinator library for Scala

Many (but not all) of the other parser combinator libraries use similar names/symbols for the combinators

scala-parser-combinatorsimport scala.util.parsing.combinator._

object BasicCombinators extends Parsers

with RegexParsers {

def pA : Parser[Char] = 'A' def pB : Parser[Char] = 'B' def pAlt: Parser[Char] = pA | pB def pSeq: Parser[Char ~ Char] = pA ~ pB def pRep: Parser[List[Char]] = rep(pA) def pOpt: Parser[Option[Char]] = opt(pA)}

import scala.util.parsing.combinator._


with RegexParsers {


scala-parser-combinatorsBase type



with RegexParsers {



Useful extensions



with RegexParsers {



alternative, sequence, repetition, optionals ...


trait Parsers { type Input

sealed trait ParseResult[+T]

trait Parser[+T] extends (Input => ParseResult[T])}





Success[+T] | NoSuccess...plus the input not consumed






~ Input => (T, Input)




trait Parser[+T] extends (Input => ParseResult[T])} A Parser is a function taking some input and

returning either some value of type T, or a failure


Returning to our language

1;

2+2;

3*3+4*4*(1+2);

Returning to our languageobject Parser1 extends RegexParsers {

def pL1: Parser[List[Int]] = rep(pExpr <~ ";")

def pExpr = pAdd

def pAdd: Parser[Int] = pProd ~ "+" ~ pAdd ^^ { case left ~ _ ~ right => left + right } | pProd

def pProd: Parser[Int] = pAtom ~ "*" ~ pProd ^^ { case left ~ _ ~ right => left * right } | pAtom

def pAtom: Parser[Int] = """\d+""".r ^^ (_.toInt) | "(" ~> pExpr <~ ")"}



def pExpr = pAdd




Many things going on



def pExpr = pAdd




We are evaluating the result directly, avoiding construct an AST



def pExpr = pAdd




Precedence rules are encoded in the grammar



def pExpr = pAdd




This is evaluating from right to left.

If you need to control associativity, the rep combinator is your friend.



def pExpr = pAdd




Regular expressions are converted to Parsers implicitly



def pExpr = pAdd




^^ is an alias of the map function



def pExpr = pAdd




Allows to convert the Parser[String] to Parser[Int]^^ is an alias of

the map function



def pExpr = pAdd




Allows to convert the Parser[String] to Parser[Int]

String => Int

^^ is an alias of the map function



def pExpr = pAdd




Parse two inputs in sequence discarding the input of the left or the right respectively



def pExpr = pAdd




Pattern matching anonymous function



def pExpr = pAdd





Allows to deconstruct the result of this parser



def pExpr = pAdd





Allows to deconstruct the result of this parser

Int IntString

“+” discarded Int


def pL1: Parser[List[Int]] = rep(pExpr <~ ";") def pExpr = pAdd




We want a list of all the expressions we encounter separated by “;”


def pL1: Parser[List[Int]] = rep(pExpr <~ ";") def pExpr = pAdd




All good?

Mini testimport Parser1._

parse(pExpr, "1") //> [1.2] parsed: 1parse(pExpr, "2+2") //> [1.4] parsed: 4parse(pExpr, "3*3+4*4*(1+2)") //> [1.14] parsed: 57

parse(pL1, """1; |2+2; |3*3+4*4*(1+2);""".stripMargin) //> [3.15] parsed: List(1, 4, 57)

parse(pL1, "1; 2+2; 3*3+4*4*(1+2);") //> [1.8] parsed: List(1, 4, 57)

Now …

Now …

What if we want to add variables?


We need to handle two more cases:

Assignation of variablesReferences to variables




def pRef = pName ^^ { x => get(symtab, x)}

def pVar = "var" ~ pName ~ "=" ~ pExpr ^^ { case _ ~ name ~ _ ~ expr => put(symtab, name, expr) }



And we need to carry a symbol table ...



?def pRef = pName ^^ { x => get(symtab, x)}

Carrying a symbol table

On pRef we need to access the symbol table but leave it unmodified

On pVar we need to update the symbol table

Otherwise we just need to pass it untouched

def pRef = pName ^^ (x => get(symtab, x))


Enter scalaz.State[S,A]

scalaz has a type that represents state transitions

State[S,A] wraps functions with type:

S => (S, A)Current state

New stateValue computed


We need to change the type of our parsers ...


We need to change the type of our parsers ...

Parser[A]

Parser[State[S,A]]

Parser[State[SymTab, A]]

Parser[S[A]]

P[A]


We need to change the type of our parsers ...We now return A, but in the context of some stateful computationParser[A]

Parser[State[S,A]]


Parser[S[A]]

P[A]


We need to change the type of our parsers ...We now return A, but in the context of some stateful computation

Our state is the symbol table~Map[String, Int]

Parser[A]

Parser[State[S,A]]


Parser[S[A]]

P[A]

Parser[A]

Parser[State[S,A]]


Parser[S[A]]

P[A]



type S[A] = State[SymTab,A]


Parser[A]

Parser[State[S,A]]


Parser[S[A]]

P[A]



type S[A] = State[SymTab,A]


type P[A] = Parser[S[A]]


… and change the parsers themselves a bit


… and change the parsers themselves a bitdef pAdd: P[Int] = pProd ~ "+" ~ pAdd ^^ { case leftS ~ _ ~ rightS => for { left <- leftS right <- rightS } yield { left + right } } | pProd





P[Int] == Parser[S[Int]]





S[Int]





S[Int]

leftS and rightS areS[Int] and can’t be added directly





S[Int]

But State[S,A] is a Monad[A]

State[SymTab,A]





S[Int]

State[SymTab,A]So we can use a for-comprehension to work with the ints





S[Int]

State[SymTab,A]When we treatState[S,A] as a Monad[A], we work with the values, and the state transitions happens in the background





S[Int]

State[SymTab,A]Here we are chaining three state transitionss0 => (s1, left)s1 => (s2, right)s2 => (s2, left+right)





S[Int]

State[SymTab,A]Which can be seen as one big transition:

s0 => ... (s2, left+right)

The State[S,A] we return


… and change the parsers themselves a bitdef pRef: Parser[Int] = pName ^^ { x => get(symtab, x) }

def pRef: P[Int] = pName ^^ { name => for { symtab <- State.get } yield { symtab(name) } }




Here we need to access to the symbol table—our state




The trick of State.get is to create a new State[S,S] where the state is used as the value




The trick of State.get is to create a new State[S,S] where the state is used as the value

Wraps a state transition { s => (s, s) }





With such State[S,S] created, now we can use again a for-comprehension to work with the symbol table (the value)





We are chaining two state transitions:s => (s, s)s => (s, s(name))

… and change the parsers themselves a bit


def pRef: Parser[Int] = pName ^^ { x => get(symtab, x) }



Which can be seen as:

s => ... (s, s(name))

The State[SymTab, Int] == S[Int] we return


… and change the parsers themselves a bitdef pVar: Parser[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ expr => put(symtab,name,expr) }

def pVar: P[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <-State.put( symtab+(name->expr) ) } yield { unit } }




The last case




We need to modify the state





S[Int]





S[SymTab]S[Int]




To change the state we use State.put

S[SymTab]S[Int]



Given s, it creates aState[S,Unit] representing this state transition: { _ => (s, ()) }


S[SymTab]S[Int]



This discards any previous state in favor of the one provided, and returns () { _ => (s, ()) }


S[SymTab]S[Int]




So, the value we get in the for-comprehension is the unit

S[Unit]

S[SymTab]S[Int]




We have four state transitions:s0 => (s1, expr)s1 => (s1, s1)_ => (s2=s1+(name->expr), ())s2 => (s2, ())

S[Unit]

S[SymTab]S[Int]




So we are returning:

s0 => … (s1+(name->expr), ())

State[SymTab, Unit] == S[Unit] S[Unit]

S[SymTab]S[Int]




Now we know the basics of the State monad

Running the stateimport Parser2._

parse(pExpr, "1") //> [1.2] parsed: scalaz.package$State$$anon$3@7f560810



Our parser now returns State[SymTab,A], not directly a value



To compute the final value, we need to provide an initial state and then all the state transitions can be run


parse(pExpr, "1").map(state => state.run(Map.empty))//> [1.2] parsed: (Map(),1)





map over ParseResult[S[Int]]




map over ParseResult[S[Int]]Initial state




map over ParseResult[S[Int]]Initial state

We obtain the final state and the value computed


parse(pExpr, "1").map(_.run(Map.empty))//> [1.2] parsed: (Map(),1)

parse(pVar, "var x = 1").map(_.run(Map.empty))//> [1.10] parsed: (Map(x -> 1),())

parse(pExpr, "y * 3").map(_.run(Map("y" -> 2)))//> [1.6] parsed: (Map(y -> 2),6)





The final state reflects the binding of 1 to x





The final state reflects the binding of 1 to x

We can provide bindings in the initial state

Running the stateparse(pL2, """var x = 1; |var y = x * 3; |x + y; |(x + 1) * 3 + 1; |var z = 8;""".stripMargin).map(_.run(Map.empty))//> [5.11] parsed: (Map(x -> 1, y -> 3, z -> 8),List(4, 7))

Now ...

Now ...

What if we reference an undeclared variable?parse(pL2, "var x = y;").map(_.run(Map.empty))

Now ...

What if we reference an undeclared variable?parse(pL2, "var x = y;").map(_.run(Map.empty))

Exception in thread "main" java.util.NoSuchElementException: key not found: y

at scala.collection.MapLike$class.default(MapLike.scala:228)at scala.collection.AbstractMap.default(Map.scala:59)at scala.collection.MapLike$class.apply(MapLike.scala:141)at scala.collection.AbstractMap.apply(Map.scala:59)...

Handling errors

If our computation can fail, then our types are wrong. We can’t just return A

Handling errors

If our computation can fail, then our types are wrong. We can’t just return A

We need to handleReferences to undeclared variablesDeclarations of already declared variables

scalaz.\/

\/[A,B] == A\/B, called “or”, “xor”, “either”, “disjoint union”,..., can be used to handle the cases when a computation can fail

A \/ BThe type of the error. By convention on the left

The type of the value we are computing. By convention on the right

scalaz.\/

\/[A,B] is the scalaz version ofscala.Either[A,B]

Handling errors

We will have to change our types again:

type V[A] = Err \/ A

type S[A] = State[SymTab, V[A]]

sealed abstract class Err(override val toString: String)

case class NotDeclared(name: String) extends Err(s"`$name' not declared.")

case class AlreadyDeclared(name: String, value: Int) extends Err(s"`$name' already declared with value `$value'.")

Bad news

Bad news

It gets ugly ...

It gets ugly ...def pVar: P[Unit] = "var" ~ pName ~ "=" ~ pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { exprV <- exprS symtab <- State.get[SymTab] newSymTab = for { expr <- exprV s <- symtab.get(name) match { // if the name exists, creates a left with the error case Some(v) => AlreadyDeclared(name, v).left // if not exists, returns the symtab as a right case None => symtab.right } } yield { s + (name -> expr) // only executes if everything is right } res <- newSymTab match { // if the newSymTab is a left, I need to put the error in the State case -\/(l) => State.state[SymTab, V[Unit]](l.left) // if the newSymTab is a right, I need to use it as the new state, and put the () inside a \/ case \/-(r) => State.put(r).map(_.right) } } yield { res }}


This is just one parser


exprS now contains a \/[Err,Int]


I’m using an extra for-comprehension to deal with it


Some juggling to construct a left value if the symbol table already has the name


If either exprV or this contains a left, all the for-comprehension will be a left, ie, newSymTab will be a left


Here I’m “folding” over the\/[Err, SymTab]. If is a left, I need to put that inside the State. If is a right, I need to use the value as the new state


-\/ is the extractor of left values, \/- is the extractor of right values


State.state allows to put a value inside a State[S,A]. For a given `a’, represents a transition { s => (s, a) }


...

Good news

Good news

Monad transformers

Monad transformers

The only thing I know about them, is that they let you combine the effects of two monads, ...

Monad transformers


… scalaz has a monad transformer, StateT, that let you combine the State monad with any other monad, …

Monad transformers


… scalaz has a monad transformer, StateT, that let you combine the State monad with any other monad, …

… and this solves all our problems

Combining State and \/

This are our final type aliases

type SymTab = Map[String, Int]


type S[A] = StateT[V, SymTab, A]








This is the only new one







StateT is parameterized on the monad with which is combined







Here is using the monad of\/[A,B]


And one final ingredientval State = StateT.stateTMonadState[SymTab, V]



The companion object of StateT does not have the functions get, put, state, etc.



Instead we have to instantiate one of this “beasts”

Combining State and \/With that changes, our parser works as before without any modification

Combining State and \/import Parser4._

parse(pExpr, "1").map(_.run(Map.empty))//> [1.2] parsed: \/-((Map(),1))

parse(pVar, "var x = 1").map(_.run(Map.empty))//> [1.10] parsed: \/-((Map(x -> 1),()))

parse(pExpr, "y * 3").map(_.run(Map("y" -> 2)))//> [1.6] parsed: \/-((Map(y -> 2),6))

Combining State and \/parse(pL4, """var x = 1; |var y = x * 3; |x + y; |(x + 1) * 3 + 1; |var z = 8;""".stripMargin).map(_.run(Map.empty))//> [5.11] parsed: \/-((Map(x -> 1, y -> 3, z -> 8),List(4, 7)))

Combining State and \/parse(pL4, """var x = 1; |var y = x * 3; |x + y; |(x + 1) * 3 + 1; |var z = 8;""".stripMargin).map(_.run(Map.empty))//> [5.11] parsed: \/-((Map(x -> 1, y -> 3, z -> 8),List(4, 7)))

The only difference is that now the results are inside a \/, in particular a \/-, ie, a right value, which indicates that everything is OK.

Handling errors

Everything else is working the same, even our exceptions when an undeclared variable is referenced

Handling errors

Everything else is working the same, even our exceptions when an undeclared variable is referenced

We must change pRef y pVar to handle the error cases

Handling errorsdef pRef: P[Int] = pName ^^ { name => for { symtab <- State.get } yield { symtab(name) } }

def pRef: P[Int] = pName ^^ { name => for { symtab <- State.get vErrInt = symtab.get(name) .toRightDisjunction( NotDeclared(name)) res <- StateT[V, SymTab, Int] { s => vErrInt.map(s -> _) } } yield { res } }



We are converting an Option[Int] into \/[Err,Int]



We construct our StateT “by hand” by passing the function directly



This StateT wraps functions of type SymTab => V[(SymTab,A)] == SymTab => \/[Err, (SymTab,A)]



vErrInt is of type\/[Err, Int], we convert it to\/[Err, (SymTab, Int)] with map



This state transition will only be made if vErrInt is a right value. If not, all the computation will return the left value



Here is happening the combination of both monad effects

Handling errorsdef pVar: P[Unit] = "var" ~ pName ~ "=" ~ pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <- symtab.get(name) match { case Some(v) => StateT[V, SymTab, Unit] { s => AlreadyDeclared(name, v).left } case None => State.put(symtab + (name->expr)) } } yield { unit }}



def pVar: P[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <-State.put( symtab+(name->expr) ) } yield { unit } } We check if the declaration

already exists


def pVar: P[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <-State.put( symtab+(name->expr) ) } yield { unit } } If exists, we construct a StateT with

a function returning a left. This stops any further state transitions


def pVar: P[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <-State.put( symtab+(name->expr) ) } yield { unit } } If the value not exists, we proceed as

before, updating the state


def pVar: P[Unit] = "var"~pName~"="~pExpr ^^ { case _ ~ name ~ _ ~ exprS => for { expr <- exprS symtab <- State.get unit <-State.put( symtab+(name->expr) ) } yield { unit } } In both cases we are returning

StateT[V, SymTab, Unit]

Now errors are handled properlyimport Parser4._

parse(pL4, "var x = y;").map(_.run(Map.empty))//> [1.11] parsed: -\/(`y' not declared.)

parse(pL4, "var x = 1; x * x; var x = 2;").map(_.run(Map.empty))//> [1.29] parsed: -\/(`x' already declared with value `1'.)

Now errors are handled properlyimport Parser4._

parse(pL4, "var x = y;").map(_.run(Map.empty))//> [1.11] parsed: -\/(`y' not declared.)

parse(pL4, "var x = 1; x * x; var x = 2;").map(_.run(Map.empty))//> [1.29] parsed: -\/(`x' already declared with value `1'.)

Both return left values

github.com/gerferra/parser-stateT

FIN

transform your state \/ err

Software