functional operations - susan potter
TRANSCRIPT
Functional Operations
#dmconf1521 November 2015
Susan PotterLookout
twitter: @SusanPottergithub: mbbx6spp
% whoami
Figure: From developer to (dev)ops engineer
Agenda
1 Motivation for reasoning
2 Review functional programming 101
3 Illustrate forms of reasoning
4 Case study: Reimagine package management
Edsger Dijkstra
Reliability
“Those who want really reliable software will discoverthat they must find means of avoiding the majority ofbugs to start with, and as a result the programmingprocess will become cheaper. If you want more effectiveprogrammers, you will discover that they should notwaste their time debugging, they should not introducethe bugs to start with.”[?]
Why care now?
1 Economic factorsnecessity of distributed systems
2 Human factorshigh churn/turnover, low quality of ops life
3 Technological factorsprogrammable infrastructure & FP no longer just for academics
Why care now?
1 Economic factorsnecessity of distributed systems
2 Human factorshigh churn/turnover, low quality of ops life
3 Technological factorsprogrammable infrastructure & FP no longer just for academics
Why care now?
1 Economic factorsnecessity of distributed systems
2 Human factorshigh churn/turnover, low quality of ops life
3 Technological factorsprogrammable infrastructure & FP no longer just for academics
The Problem. . .
Application Delivery
1 Provision infrastructure
2 Configure nodes
3 Orchestrate services
Need to support more
1 Application services
2 Environments
3 Data services
4 Distributed services
Optimize for
1 Scalability solved by on-demand ”cloud”
2 Reliability solved by . . .
Optimize for
1 Scalability solved by on-demand ”cloud”
2 Reliability solved by . . .
So what yields reliability?
Reason
The required techniques of effective reasoning are prettyformal, but as long as programming is done by peoplethat don’t master them, the software crisis will remainwith us and will be considered an incurable disease. [?]
Functions 101
Functions have inputs (Ruby)
1 # Two input arguments here
2 def add(x, y)
3 x + y
4 end
5
6 # One input argument here
7 def len(s)
8 s.size
9 end
Functions have inputs (Scala)
1 object Functions {
2 // Two typed input arguments here
3 def add(x: Int , y: Int) = x + y
4
5 // One typed input argument here
6 def len(s: String) = s.size
7 }
Functions return a result
1 scala > add(5, 6)
2 res0: Int = 11
3
4 scala > len("Hello ,␣Barcelona")
5 res1: Int = 16
Only depend on inputs 1/2
1 scala > val defaultTimeout: Int = 30
2 scala > val timeout1: Option[Int] = Some (15)
3 scala > val timeout2: Option[Int] = None
4 scala > :paste
5 def defaulter[A](a: => A, ma: Option[A]) =
6 ma match {
7 case Some(x) => x
8 case None => a
9 }
Only depend on inputs 2/2
1 scala > timeout1
2 timeout1: Option[Int] = Some (15)
3
4 scala > timeout2
5 timeout2: Option[Int] = None
6
7 scala > defaulter(defaultTimeout , timeout1)
8 res0: Int = 15
9
10 scala > defaulter(defaultTimeout , timeout2)
11 res1: Int = 30
Return same result given same inputs
1 scala > len("Hello ,␣Barcelona")
2 res0: Int = 16
3
4 scala > len("Hello ,␣Barcelona")
5 res1: Int = 16
6 ...
7 scala > len("Hello ,␣Barcelona")
8 res3333333333: Int = 16
9
10 scala > // Always!
The Big idea
Referential TransparencyGiven same inputs, return same result. Always.
Functions can use other values
1 // type aliasing a function
2 type Pred[A] = A => Boolean
3
4 // Passing a function as an input argument
5 def is[A](p: Pred[A])(a: A) = p(a)
6
7 // This uses already defined function +is+
8 def not[A](p: Pred[A])(a: A) = !is(p)(a)
Values can be functions
1 // Returning a function as a value
2 def lessThanN(n: Int): Pred[Int] = _ < n
3
4 // Using two in scope functions
5 def islessThanN(n: Int)(x: Int) =
6 is(ltN(n))(x)
7
8 // Those values can be functions :)
Another important idea
Higher Order FunctionsBuild useful functions from simpler functions!
Questions so far?
Figure: Awake?
Function Composition
UNIX Pipes 1/4
1 sh> echo -n "Hello ~~~" \
2 | sed ’s/~//g’ \
3 | tr ’[:lower:]’ ’[:upper:]’ \
4 | wc -c
5 5
UNIX Pipes 2/4
1 sh> alias sanitize=’sed "s/~//g"’
2 sh> alias toUpper=’tr "[: lower :]" "[: upper:]"’
3 sh> alias len=’wc -c’
4
5 sh> alias myfun0=’sanitize | toUpper ’
6 sh> alias myfun1=’myfun0 | wc -c’
UNIX Pipes 3/4
1 sh> echo -n "Hello ~~~" \
2 | sed ’s/~//g’ \
3 | tr ’[:lower:]’ ’[:upper:]’ \
4 | wc -c
5 5
6
7 sh> echo -n "Hello ~~~" | myfun1
8 5
UNIX Pipes 4/4
• Character-based
• File descriptors
Function Composition 1/3
1 def toUpper = (s: String) => s.toUpperCase
2 def len = (s: String) => s.size
3 def sanitize = "~".r replaceAllIn (_, "")
Function Composition 2/3
1 scala > def myfun0 = sanitize andThen toUpper
2 scala > def myfun1 = myfun0 andThen len
3
4 scala > myfun0 "Hello ~~~"
5 res0 = HELLO
6
7 scala > myfun1 "Hello ~~~"
8 res1: Int = 5
Function Composition 3/3
• Value-based
• Functions
Questions so far?
Figure: Awake?
Functions as building blocks
Figure: Build solid walls/foundations from simple, regular building blocks
Reasoning
Equational Reasoning
• Simpler testing; reduce test suite complexity
• Substitute repeated expression with name
• Refactoring is trivial; not error prone
Testing non-RT code
Figure: Side effecting, non-RT code requires complex testing setup.
Equational Reasoning
1 scala > f(v1, v2, otherfun) == expected
2 res0: Boolean = true
3
• no setup/teardown complexity to test assertions
• no mutation of SUT just for testing
Equational Reasoning
1 scala > val x = "YOLO"
2 x: java.lang.String = YOLO
3
4 scala > val r1 = x.reverse
5 r1: String = OLOY
6
7 scala > "YOLO".reverse
8 res0: String = OLOY
9
Equational Reasoning
1 scala > var age = 27
2 scala > def incr = { age += 1; age }
3
4 scala > incr
5 res0: Int = 28
6
7 scala > incr
8 res1: Int = 29
9
Axiomatic Reasoning
• Axioms are basic assumptions
• Allows us to specify properties
• Infer potential problems due to properties
• What can we guarantee? What can we not?
Axiomatic Reasoning
1 class Vpc < AwsResource
2 def initialize(conf)
3 @cidr = conf[’vpc.cidr’]
4 end
5 end
6 # Is the Vpc instance created with an
7 # empty conf Hash guaranteeing the
8 # premise of the initialize contract?
9
Axiomatic Reasoning
1 // validate inputs before instantiating
2 case class Vpc(cidr: Cidr)
3
4 def createVpc(ipStr: String , pfx: String) =
5 for {
6 ip <- IpAddress.toIpAddress(ipStr)
7 prefix <- Cidr.toPrefix(pfx)
8 } yield Vpc(Cidr(ip , prefix ))
9
Axiomatic Reasoning
1 // if fun is 1-to -1 or bijective
2 // we know inverse must exist
3 def toLower(u: UpperCaseChar ): LowerCaseChar = ???
4 def toUpper(l: LowerCaseChar ): UpperCaseChar = ???
5
6 // idempotency typically important in ops
7 f(f(x)) == f(x)
8
Axiomatic Reasoning
1 // commutativity important in dist sys
2 // f, binary operator over type A
3 // x and y are of type A then
4 // x @ y = y @ x
5 f(x, y) == f(y, x)
6
7 // associativity important in concurrency
8 // f, binary operator over type A
9 // x, y, and z are of type A then
10 // x @ (y @ z) = (x @ y) @ z
11 f(x, f(y, z)) == f(f(x, y), z)
12
Axiomatic Reasoning - DistributedSystems
• Simple causal consistency (partial ordering)
• CRDTs (semilattice algebra)
• Invariant confluence (coordination freeexecution)
Axiomatic Reasoning - Concurrency
• MVars, TVars, LVars have algebraic properties
• Design for liveness via axiomatic reasoning
Axiomatic Reasoning - More ExhaustiveTesting
• Property-based testing
1 // untested - uses scalacheck
2 def inverse[A]( implicit a: Action[A]) =
3 Prop.forAll { (x: A) =>
4 x === a.inverse(a.inverse(x))
5 }
6
7 implicit def actionArbitrary[A]
8 (implicit a: Arbitrary[A]) = Arbitrary { /* TODO */ }
9
Generic Reasoning
• Reasoning on generic type functions
• Less specific the types the more we know aboutthe function
• Find design-time bugs
Generic Reasoning
1 def f0[A](a: A): A = ???
2
Generic Reasoning
1 // Only definition
2 def f0[A](a: A): A = a
3
Generic Reasoning
1 def f1[A](a: A)(ma: Option[A]): A = ???
2
Generic Reasoning
1 def f1[A](a: A)(ma: Option[A]): A = a
2
1 def f1[A](a: A)(ma: Option[A]): A = ma match {
2 case Some(x) => x
3 case None => a
4 }
5
Generic Reasoning
1 def f2[A](as: List[A]): A = ???
2
Generic Reasoning
1 // what about an empty list? Yikes!
2 def f2[A](as: List[A]): A = ???
3
Generic Reasoning
1 // Assuming we want the head element!
2 // A more sensible type signature design
3 def f3[A](as: List[A]): Option[A] = as match {
4 case Nil => None
5 case a :: rest => a
6 }
7
Generic Reasoning
1 def f4[A](ma: Option[A]): A = ???
2
Review
Reasoning Process
• Assume as little as possible
• Define all inputs
• Build only on top of RT
• Derive properties from domain
• Infer properties from generic types
Methods / Techniques
• Encode normal error cases in results
• Strive for termination
• Strive for RT where ever possible
Iteration / Improvement
• So much to learn
• Start out informal; add formal reasoning asneeded/learned
• The more we do the easier it gets
Example in the wild
Mainstream Package Management
Based on shared + mutable state (filesystem)
Violates RT
Alternatives
• shared + immutable
• private + mutable
• expensive coarse grained locks
• hybrid without the expense
Define all inputs
• Force clean build env (chroot)
• Requires explicit inputs
• Full dependency definition
Ensure RT
• Use private mutable space
• Different inputs, different result
• Symlink unique results (atomic op)
Different inputs give unique results
• Shared + immutable provides RT
• New inputs result in different path
• Install many versions/configurations
• Means we can do binary substitution
RT foundation
• Update existing node more reliably
• Supports immutable infrastructure
• Supports congruent configuration
Recap
1 Referential Transparency allows us to reason
2 Functions are simple, regular building blocks
3 HOFs & Composition is the motar
4 Logical reasoning provides solid foundation
Recap
1 Referential Transparency allows us to reason
2 Functions are simple, regular building blocks
3 HOFs & Composition is the motar
4 Logical reasoning provides solid foundation
Don’t fear the maths!
Figure: Domain specific language of the sciences and engineering
Questions
Figure: Heckle me @SusanPotter later too.