scala 2013 review
DESCRIPTION
Slides from my talk at the Junction (Jan 24, 2013) Single-core performance has hit a ceiling, and building web-scale multi-core applications using imperative programming models is nightmarishly difficult. Parallel programming creates a new set of challenges, best practices and design patterns. Scala is designed to enable building scalable systems, elegantly blending functional and object oriented paradigms into an expressive and concise language, while retaining interoperability with Java. Scala is the fastest growing JVM programming language, being rapidly adopted by leading companies such as Twitter, LinkedIn and FourSquare. This presentation provides a comprehensive overview of the language, which managed to increase type safety while feeling more dynamic, being more concise and improving readability at the same time. We will see how Scala simplifies real life problems by empowering the developer with powerful functional programming primitives, without giving up on the object oriented paradigm. The overview includes tools for multi-core programming in Scala, the type system, collection framework and domain-specific languages. We’ll explore the power of compile-time meta-programming, which is made possible by the newly released Scala 2.10, and get a glimpse into what to expect from 2.11 in 2014. We will also see how Scala helps overcome the inherent limitations of Java, such as type erasure, array covariance and boxing overhead. Multiple examples emphasize how Scala pushes the JVM harder than any other mainstream language through the infinite number of boilerplate busters, increased type safety and productivity boosters from a Java developer’s perspective.TRANSCRIPT
2013 overview Sagie Davidovich
singularityworld.com linkedin.com/in/sagied
2013 JVM Languages Landscape Static
Dynamic
Functional Object O
riented
Who’s using scala
Performance study by Google
Exponential growth in demand
Thoughtworks Technology Radar 2012
Evaluate
Consider
Hold
Language Design Trade-‐‑offs
Complexity
Expressiveness Performance
Positioning
JVM
Functional
Statically Typed
Object Oriented
Prof. Martin Odersky • Designer of Java Generics • Creator of JavaC • Prof. at EPFL • ACM Fellow • Founder of
Class Employee -‐‑ Java public class Employee implements Serializable { private final String firstName; private final String lastName; public Employee (String firstName, String lastName) { this.firstName = firstName; this.lastName = lastName; } public String getFirstName() { return lastName; } public String getLastName() { return firstName; } public Employee withFirstName(String firstName) { return new Employee (firstName, lastName); } public Employee withLastName(String lastName) { return new Employee (firstName, lastName); }
public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Employee employee = (Employee) o; if (firstName != null ? !firstName.equals(employee.firstName) : person.firstName != null) {
return false; } if (lastName != null ? !lastName.equals(employee.lastName) : employee.lastName != null) { return true; } return true; } public int hashCode() { int result = firstName != null ? firstName.hashCode() : 0; result = 31 * result + (lastName != null ? lastName.hashCode() : 0); return result; } public String toString() { return "Employee(" + firstName + "," + lastName + ")"; } }
Oops, should be false Anyone noticed?
Class Employee -‐‑ Scala
case class Employee(firstName: String, lastName: String)!
• Constructor • Copy constructors • Fields • GeVers / (seVers) • equals • hashCode • toString • Recursive decomposition • Companion object
Singletons Java public class OneAndOnly { private final static OneAndOnly INSTANCE = new OneAndOnly(); private OneAndOnly() {} public static OneAndOnly getInstance() { return OneAndOnly.INSTANCE; } }
Scala
object OneAndOnly
Everything is an object
1.+(2) Same as
1 + 2
Every block returns a value (no statements, only expressions)
def add (x:Int, y:Int) = x + {val tmp = y; tmp * 3}!!if (x > 2) 3 else 4!!val doubled = for(i <- (1 to 10)) yield i * 2!
Type inference scala> val x = 3 x: Int = 3
scala> def x(y:Int) = y * 9 x: (y: Int)Int
scala> def x(y:Int) = if (y == 1) List(y) else Set(y) x: (y: Int) Iterable[Int]
Higher order functions List(1, 2, 3).map((x: Int) => x + 1)
=
List(1, 2, 3).map(x => x + 1) // type is inferred
=
List(1, 2, 3).map(_ + 1) // placeholder notation
Universal access principle import System._!!class Clz {! def w = currentTimeMillis / 1000! val x = currentTimeMillis / 1000! var y = currentTimeMillis / 1000! lazy val z = currentTimeMillis / 1000!}!
Default methods apply and unapply
object Square{ def apply(d: Double) = d * d def unapply(d: Double) = Some(math.sqrt(d)) } //apply val s = Square(3) // 9 // unaply 16 match { case Square(x) => x } // 4
Curried functions • Add new flow control and language constructs!
def loop(n: Int)(body: => Any) {! (0 until n) foreach (n => body)!}!!
loop(2) {! println("IM IN YR LOOP!")!}!
• Multiple varargs lists are possible!!def crossProduct[L,R](left: L*)(right: R*) = for (l <- left; r <- right) yield (l,r)!crossProduct(1,2,3)(‘a’,’b’,’c’)!
• Partially applied functions!!!
Multiple inheritance without the diamond problem
Type system
Strong
Static + Dynamic
Nominal + Structural
Explicit + Inferred
Erasure
Co/Contra-‐‑variant
Value + Reference
Nonnullable Monads
Case classes Path dependent
Anonymous
Self types
Type aliases
Functional
Existential Implicit
Traits
Type system
Type safety (Java example) // compiles fine in Java (arrays are co-variant!!)
int[] ints = {3}; Object[] objects = ints; // ArrayStoreException in runtime
objects[0] = new Object();
Structural types object Closer { def using(closeable: { def close(): Unit }, f: => Unit) { try { f } finally { closeable.close } } }
Immutable Collections
Mutable Collections
Simpler is safer Indexing by first character
Java List<String> keywords = Arrays.asList("Apple", "Banana", "Beer"); Map<Character, List<String>> result = new HashMap<Character, List<String>>(); for(String k : keywords) { char firstChar = k.charAt(1); if(!result.containsKey(firstChar)) { result.put(firstChar, new ArrayList<String>());
} result.get(firstChar).add(k); } for (List<String> list : result.values()) { Collections.sort(list); } Scala val keywords = List("Apple", "Banana", "Beer”) val result = keywords.sorted.groupBy(_.head)
Oops, here’s a bug. Anyone noticed?
Another example !Java!public List<String> empNames(ArrayList<Employee> employees) {!
!ArrayList<String> result = new ArrayList<String>();!!for (Employee emp: employees) {!! !result.add(emp.getName());!!}!!return result;!
}!!!!Scala!def empNames(employees: List[Employee]) = employees map getName!
Map combinator paVern !Java!public List<String> empNames(ArrayList<Employee> employees) {!
!ArrayList<String> res = new ArrayList<String>();!!for (Employee emp: employees) {!! !res.add(emp.getName());!!}!!return res;!
}!!!!Scala!def empNames(employees: List[Employee]) = employees map getName!
Really need to encapsulate??
Another example !Java!public static int factorial(int n) {! int res = 1;! for (int i = 1; i <= n; i++) {! res *= i;! }! return res;!
}!!!
Scala!def factorial(n: Int) = (1 to n).reduce(_*_)!
Reduce combinator paVern !Java!public static int factorial(int n) {! int res = 1;! for (int i = 1; i <= n; i++) {! res *= i;! }! return res;!
}!!!
Scala!def factorial(n: Int) = (1 to n).reduce(_ * _)!
Combinatorics o (1 to 5) combinations 2
List(Vector(1, 2), Vector(1, 3), Vector(1, 4), Vector(1, 5), Vector(2, 3), Vector(2, 4), Vector(2, 5), Vector(3, 4), Vector(3, 5), Vector(4, 5))
o (1 to 5).permutations "George W. Bush".split(" ").permutations List(Array(George, W., Bush), Array(George, Bush, W.), Array(W., George, Bush), Array(W., Bush, George), Array(Bush, George, W.), Array(Bush, W., George))
o "George W. Bush".split(" ").toSet.subsets List(Set(), Set(George), Set(W.), Set(Bush), Set(George, W.), Set(George, Bush), Set(W., Bush), Set(George, W., Bush))
Collection framework ++ ++: +: /: /:\ !:+ :: ::: :\ addString !aggregate andThen apply applyOrElse asInstanceOf !canEqual collect collectFirst combinations companion !compose contains containsSlice copyToArray copyToBuffer !
corresponds count diff distinct drop !dropRight dropWhile endsWith exists filter !filterNot find flatMap flatten fold !foldLeft foldRight forall foreach genericBuilder !groupBy grouped hasDefiniteSize head headOption !indexOf indexOfSlice indexWhere indices init !inits intersect isDefinedAt isEmpty isInstanceOf !isTraversableAgain iterator last lastIndexOf lastIndexOfSlice !
lastIndexWhere lastOption length lengthCompare lift !map mapConserve max maxBy min !minBy mkString nonEmpty orElse padTo !par partition patch permutations prefixLength !product productArity productElement productIterator productPrefix !reduce reduceLeft reduceLeftOption reduceOption reduceRight !reduceRightOption repr reverse reverseIterator reverseMap !reverse_::: runWith sameElements scan scanLeft !
scanRight segmentLength seq size slice !sliding sortBy sortWith sorted span !splitAt startsWith stringPrefix sum tail !tails take takeRight takeWhile to !toArray toBuffer toIndexedSeq toIterable toIterator !toList toMap toSeq toSet toStream !toString toTraversable toVector transpose union !unzip unzip3 updated view withFilter !zip zipAll zipWithIndex !
Placeholder syntax for anonymous functions
_ + 1 x => x + 1 _ * _ (x1, x2) => x1 * x2
(_: Int) * 2 (x: Int) => x * 2 if (_) x else y z => if (z) x else y
_.map(f) x => x.map(f)
_.map(_ + 1) x => x.map(y => y + 1)
Regular Placeholder
Distributed & multicore computing
• Parallel collections (myList.par.sum) • Futures & promises • Actors • STM – makes Java heap ACID compatible
• Hadoop • Spark & Storm • Plain old Java threads • Scalding – Twitter’s Scala based MapReduce
implementation
Core
3rd P
arty
Calling Hadoop from Scala
Java public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } }
Scala def map(key: LongWritable, value: Text, output: OutputCollector[Text, IntWritable], reporter: Reporter) = value split " " foreach (output collect (_, one))
Futures
val from1 = future {conn1.fetch} val from2 = future {conn2.fetch} from1 either from2 onSuccess println
Parallel collections List(1, 2, 3, 4, 5).par filter (_ % 2 == 0) // ParVector(2, 4)
Spark val file = spark textFile "hdfs://...” !file.flatMap(_ split " ") .map(word => (word, 1)) .reduceByKey(_ + _)!!
Tail recursion optimization
Recursion in Java has an Achilles’ heel: the call stack def factorial(n: Int, res: Long = 1): Long = if(n == 0) res else factorial(n - 1, (res * n))
Safer string composition • Standard string composition:
“I’m “ + name + “, “ + age + “ years old”
• Formatted: “I’m %s, %d years old”.format(name, age) java.util.IllegalFormatConversionException
• Interpolated: s“I’m $name, $age years old”
Create your own interpolations
xml”””<body>
<a href = “http://…”> $text</a> </body> ”””
Expression simplification – Java public Expr simplify(Expr expr) { if (expr instanceof UnOp) { UnOp unOp = (UnOp) expr; if (unOp.getOperator().equals("-")) { if (unOp.getArg() instanceof UnOp) { UnOp arg = (UnOp) unOp.getArg(); if (arg.getOperator().equals("-")) return arg.getArg(); } } } if (expr instanceof BinOp) { BinOp binOp = (BinOp) expr; if (binOp.getRight() instanceof Number) { Number arg = (Number) binOp.getRight(); if (binOp.getOperator().equals("+") && arg.getNum() == 0) return binOp.getLeft(); } } return expr; }
PaVern matching Expression simplification -‐‑ Scala
def simplify(expr: Expr): Expr = expr match {! case UnOp("-", UnOp("-", e)) => simplify(e)! case BinOp("+", e, Number(0)) => simplify(e)! case _ => expr!}!
Value classes Extensibility minus boxing overhead!!class Meter(val v: Double) extends AnyVal {! def plus (that: Meter) = new Meter(v + that.v) !}!!
Compiler generates!
object Meter {! def plus(this: Meter, that: Meter) = new Meter(v + that.v)!}!
Compile-‐‑time Meta-‐‑programming
• Language virtualization (overloading/overriding semantics of the original programming language to enable deep embedding of DSLs),
• Program reification (providing programs with means to inspect their own code),
• Self-optimization (self-application of domain-specific optimizations based on program reification),
• Algorithmic program construction (generation of code that is tedious to write with the abstractions supported by a programming language).
Meta programming Language Operate on Type
safe Expressiveness
Runtime Performance overhead
Textual preprocessors
C, Scala Text ✗ Low No
Template systems
C++ Text ✗ Low No
Compile-‐‑time meta-‐‑programming
Haskell, Scala 2.10, Nemerle
Abstract (typed) Syntax trees
✔ High
No
Byte code manipulation
Java, Scala byte-‐‑code ✗ Medium No
“Dynamic” reflection
Ruby, Scala objects ✗ High Yes
Run-‐‑time reflection
Java, Scala objects ✗ High Yes
Scala macros • Full blown compile time meta-
programming • Access to the compiler API • Written in Scala • Hygienic • Type-safe
Macros – safe printf printf(format: String, params: Any*)!printf(“value is %d”, “boom”)!// runtime exception!java.util.IllegalFormatConversionException: d != java.lang.String!!!
Macro implementation!def printf(format: String, params: Any*): Unit = macro printf_impl!def printf_impl(c: Context)(format: c.Expr[String], params: c.Expr[Any]*): c.Expr[Unit] = ...!
Efficient Assert assert(2 + 2 == 4, "weird arithmetic") // 2 + 2 == 4 will be evaluated only if assertions were enabled
Macros – units of measure val gravityOnEarth = u(9.81, "m/s^2")!
!
val heightOfMyOfficeWindow = u(3.5, "m")!
!
val speedOfImpact = sqrt(2.0 * gravityOnEarth * heightOfMyOfficeWindow)!
!
val monthsInAYear = 10b12!
hVp://scalamacros.org/usecases/units-‐‑of-‐‑measure.html
Macros – type providers
type MySqlDb(connString: String) = macro ...!
type MyDb = Base with MySqlDb("Server=127.0.0.1;Database=Foo;”)!
val products = new MyDb().products!
products.filter(p => p.name.startsWith("foo"))! !
!!
!
!
http://scalamacros.org/usecases/type-providers.html!
Inspired by F# type providers!
Type macros Injecting async methods
class D extends Lifter {!!def x = 2!!// def asyncX = future { 2 }!
}!!val d = new D!d.asyncX onComplete {!
!case Success(x) => println(x)!!case Failure(_) => println("failed")!
}!
hVp://scalamacros.org/talks/2012-‐‑12-‐‑18-‐‑MacroParadise.pdf
More uses of macros
• Ad-hoc code style and code-smell detector • Advanced domain-specific languages • Logging with minimal overhead • Pre-compiled SQL queries • GPU optimized code (see Scalaxy, ScalaCL) • Macro annotations
o @Cached
Macro annotations class atomic extends MacroAnnotation { def complete(defn: _) = macro(“backing field”) def typeCheck(defn: _) = macro("return defn itself”) } @atomic var fld: Int
GPU compilation • Enablers: Immutability, Macros • ScalaCL Collections
OpenCL-backed collections that look and behave like standard Scala collections (work in progress).
• ScalaCL Compiler Plugin optimizes Scala programs at compile-time, transforming regular Scala loops into faster code and transforming Scala functions given to ScalaCL Collections into OpenCL kernels.
An extremely hack-‐‑able compiler > scala -Xshow-phases! phase name id description! ---------- -- -----------! parser 1 parse source into ASTs, perform simple desugaring! namer 2 resolve names, attach symbols to named trees!packageobjects 3 load package objects! typer 4 the meat and potatoes: type the trees! patmat 5 translate match expressions!superaccessors 6 add super accessors in traits and nested classes! extmethods 7 add extension methods for inline classes! pickler 8 serialize symbol tables! refchecks 9 reference/override checking, translate nested objects! selectiveanf 10 ANF pre-transform for @cps! selectivecps 11 @cps-driven transform of selectiveanf assignments! uncurry 12 uncurry, translate function values to anonymous classes! tailcalls 13 replace tail calls by jumps! specialize 14 @specialized-driven class and method specialization! explicitouter 15 this refs to outer pointers, translate patterns! erasure 16 erase types, add interfaces for traits! posterasure 17 clean up erased inline classes! lazyvals 18 allocate bitmaps, translate lazy vals into lazified defs! lambdalift 19 move nested functions to top level! constructors 20 move field definitions into constructors! flatten 21 eliminate inner classes! mixin 22 mixin composition! cleanup 23 platform-specific cleanups, generate reflective calls! icode 24 generate portable intermediate code! inliner 25 optimization: do inlining!inlinehandlers 26 optimization: inline exception handlers! closelim 27 optimization: eliminate uncalled closures! dce 28 optimization: eliminate dead code! jvm 29 generate JVM bytecode! terminal 30 The last phase in the compiler chain!
Type-‐‑safe failures try { 2 / x } catch { case e:Throwable => 0} in 2.10: Try (2 / 0) // returns Failure Try (2 / 1) // returns Success Try (2 / x) getOrElse (0) List(0,1,0,2).map (x=> Try(2 / x)) filter(_.isSuccess) // List(Success(2), Success(1))
Implicit parameters implicit val db = getDB!def store(employee: Employee)(implicit db: DB)!!store(emp1oyee1) //db is passed implicitly!
Implicit methods implicit def toEuro(x: Currency): Euro = …!!def transfer(amount: Euro) {! println(”transferred” + amount)!}!!// dollar is automatically converter to euro!transfer(Dollar(3.0))!transferred 2.25 Euros!
Implicit classes !implicit class StringExtensions(s: String) {! def words = s split " "!}!!!“hello world”.words // Array(hello, world)!
Environments
ublime -‐‑ Through Ensime
-‐‑ Formally supported by TypeSafe
-‐‑ Currently the most mature
Scala REPL
DSLs (accounting) Rules to calculate an employee's paycheck:! employee's gross salary for 2 weeks! minus deductions for! federalIncomeTax, which is 25% of gross! stateIncomeTax, which is 5% of gross! insurancePremiums, which are 500 in gross’s currency! retirementFundContributions are 10% of gross!!
hVp://ofps.oreilly.com/titles/9780596155957/DomainSpecificLanguages.html
DSLs (JSON) ("person" ->! ("name" -> "Joe") ~! ("age" -> 35) ~! ("spouse" ->! ("person" ->! ("name" -> "Marilyn") ~! ("age" -> 33)! )! )!)!
DSL (XPath / json)
val titles = xml \ "channel" \ "item" \ "title“ val titles2 = json \ "feed" \ "entry" \ "title" \@ "$t"
github.com/razie/snakked
DSLs testing frameworks
val emptyStack = new Stack[String] evaluating { emptyStack.pop() } should
produce [NoSuchElementException]
Full access to the compiler import tools.reflect.ToolBox!import reflect.runtime.{currentMirror => cm}!!val tb = cm.mkToolBox()!!scala> val tree = tb parse "(5 + 9) * 3"!tree: tb.u.Tree = 5.$plus(9).$times(3)!!scala> tb eval tree!res1: Any = 42!
Reflection
val take = typeOf[List[_]].member(TermName("take")).asMethod!!reflect(List(1,3,2,4)).reflectMethod(take)(2) // List(1,2)!
Testing • ScalaCheck
o Auto-generation of inputs and mocks. o Composable check units o Inspired by Haskell QuickCheck
• ScalaTest o TDD, BDD, FeatureSpec o Selenium DSL
• Specs2 o TDD, BDD, Datatables support o Integration with Mockito, EasyMock and JMock o "hello world" must be matching("h.* w.*")
• JUnit
ScalaCheck (inspired by Haskell QuickCheck)
scala> val sqrt = forAll {(n: Int) => math.sqrt(n*n) == n } scala> propSqrt.check ! Falsified after 1 passed tests: > -1
BDD with ScalaTest describe("A Stack") { it("should throw NoSuchElementException if an empty stack is popped") in { val emptyStack = new Stack[String] evaluating { emptyStack.pop() } should produce [NoSuchElementException] } it("should pop values in last-in-first-out order") { … } }
How to learn
• twitter.github.com/scala_school • Effective Scala • Coursera Scala course (50K students last year) • Simply scala (interactive learning) • Programming in scala book • Stackoverflow
Contributing docs.scala-‐‑lang.org/sips
Contributing github.com/scala
Things to be aware of ✗ Scala Compiler has to do much more => slower
✗ Tooling is not perfect (debugging, synthetic frames, macros support) Getting better every day.
✗ Language is rapidly evolving. Deprecations should be expected.