John Nestor and Dragos Manolescu

{jnestor,dmanolescu}@whitepages.com

Whitepages Practical Experience Converting from Ruby to Reactive

August 13, 2014

Disclaimer

The following material is subjective and represents the

views of the presenters.

We hope our views and experience may be of use to

others making the transitive from Ruby to Scala, but

your mileage may vary.

Whitepages Typesafe Case Study

•WhitePages Rebuilds Core Parts of Application Stack with

Scala and Akka to Improve Scaling.

•http://j.mp/wpscala

Outline•Background

•Technology

•People

Background

About Whitepages

•Top web and mobile site for finding phones, people and

locations.

•50M unique users per month.

•35M search queries per day.

•70 engineers, mostly in Seattle.

Why replace Ruby?

Problems with Ruby legacy systems:

• High latencies.

• High resource consumption: memory and processors.

• Poor tolerance of upstream degradation:

– Limited concurrency.

– Worker starvation when blocking on slow

upstream.

– Poor connection management and healing.

Why replace Ruby with Scala?

• Elegant blend of Functional and OO paradigms.

• Static type system:

• Type inference avoids the excess boilerplate of Java.

• Compiler catches many errors.

• Runs on the JVM:

– Fast.

– Near seamless interop with JVM libraries.

– Reasonably mature tooling support.

• Actor-based concurrency framework: Akka.

Whitepages Reactive Services

• Service-oriented-architecture where communication is

Thrift or Json over HTTP.

• Both latency and throughput are very important.

• We require excellent logging and monitoring.

• Agility of development, testing, build and deployment

provide leverage and competitive advantage.

Performance

Results

Service p50 ms p99 msThroughput

RPS/core

DirSvc - Scala 25 300 80

DirSvc - Ruby 140 1200 7

Status of Reactive Scala Services

• We have been using Scala for a little more than 1.5 years.

• We started with 1 development team, we now have 4.

• We started with a single back end service. We now have 4

services in production and another 6 in development.

• Our first project had 6 Scala developers, we now have over 20

Scala developers.

• We changed not only language but also tools (more automation)

and processes (more agile).

Future Reactive Work

• Transition additional Ruby back-end services to Scala.

• Exploring new technologies:

• Spark and Mesos.

• A messaging system based on Kafka.

• A graph store based on Titan/Cassandra.

• A deployment and lifecycle-management system based

on Docker that uses remote actors.

Technology

•GitHub Enterprise

•IntelliJ

•Jenkins (for builds)

•Artifactory

•Yourkit

•SCoverage

•Scrooge (for thrift, from

Twitter)

•Scala

•Akka

•Spray

•SBT – with custom plugins

•Typesafe Config

•Scala Test

•Coda Hale Metrics

•Persist Json

Technology Selection

Whitepages Service Framework• Configuration (based on Typesafe Config, with multi-level overrides).

• Monitoring (central actor that uses Coda Hale metrics and publishes to

Graphite, Nagios and Riemann).

• Logging (custom system with central actor, Json log content and real time

streaming to external analytics).

• Admin REST API.

• REST Thrift/JSON Server (based on Spray).

• Clients (Spray, JDBC, Redis, SOLR, Riak,… -- advanced error detection/

recovery).

Technology Successes• Scala concise syntax. – Higher developer productivity.

• Use of async future based code. – Great performance.

• Immutable collections and functional programming. – Fewer bugs.

• Strong type checking. – Fewer bugs and more maintainable code (But meta-

programming becomes more difficult).

• Concurrency. – No race conditions or deadlock on complex concurrent system.

• Spray. – Great performance and an async API.

• SBT. – Able to easily customize to our needs (after steep learning curve).

• IntelliJ IDEA. – Scala support is good and keeps getting better.

• Typesafe Developer support contract. – Fast response, great answers to complex

questions. (We should have gotten it earlier!)

Technology Problems -1

• Poor documentation:

• SBT is the poster child.

• We still often have to read source code of Scala and Akka

libraries.

• Stable APIs:

• Typesafe has made great strides here.

• Other components are still troublesome.

• We lost many weeks as a result of breaking changes in Scala Test.

• We currently can’t compile a 2.11 version of Kafka because the

Zinc Scala compiler is not backward compatible.

Technology Problems -2

• Performance analysis of concurrent constructs:

• We went with the Typesafe Console.

• It never fully worked and then was abandoned.

• There are some new tools, but we have not had time to

evaluate.

Technology Problems -3

• Ecosystem:

• Not as mature as the Java ecosystem and often missing Scala

specific features. For example, Scala Mock does not support

named parameters.

• Some needed components were missing: until recently code

coverage.

• In some cases there were too many options: over 10 Json

libraries.

• Lots of Scala GitHub projects. Quality is variable; many

become abandoned.

Technology Problems -4

• Debugging:

• Debuggers work on the Java byte code, that for some Scala

constructs is quite convoluted.

• Debugging chains of futures is much harder than debugging

nested calls. Some kind of tracing is desperately needed.

• Actors have lots of power but can be exceptionally hard to

debug. We need both better approaches to building debug-

able actors and better debugging tools.

Technology Problems -5

• Language and Library Issues:

• Type erasure is a major flaw in the Scala language. It causes

lots of developer confusion and errors. The workarounds are

awkward at best.

• The lack of type checking on Actor constructors is an odd hole

in the type checking and a real annoyance.

• Some Scala code that looks simple and intuitive creates major

difficulties when a developer needs to understand what is

really going on. Implicits are one of many problems here.

Technology Overall

• The problems mentioned above create real day to day difficulties.

• In many cases Typesafe has been actively working to

incrementally correct some of these problems.

• Since we started using Scala, the number of problems has

significantly decreased and we expect that trend to continue.

• Overall, Scala/Akka is a great technology for building

reactive systems and its benefits far outweigh the

problems.

People

Levels of Scala Expertise

• Basic OO features. Easy transition from most other languages

including Java and Ruby.

• Adding functional. Ruby has functional features that are similar.

But knowing when best to use functional versus OO requires

experience.

• Adding concurrency. Via futures, promises, and actors. A whole

new world for Ruby programmers that requires a new way of

thinking.

• API design. Not all developers need to design reusable APIs, but

those that do need a higher level of experience and feature

mastery to do it well.

Building Scala Development Teams

• The are still not a large number of experienced Scala developers.

• Those out there are in considerable demand.

• We have followed a mixed strategy of both recruiting Scala

developers and also training existing Ruby developers to become

Scala developers.

• Having an experienced Scala developer greatly accelerated the

training of Ruby developers.

Note: Using Scala, Akka and other advanced technologies makes

Whitepages very attractive to some of the very best developers.

Recruiting: Finding Developers

• Personal contacts. The best way to find great developers.

• Meet-ups and Events

• John Nestor is the organizer for the Seattle Scala meet-up.

• Whitepages hosts the Scala meet-up (and several others)

• Whitepages sponsors many other technology conferences and

events.

• Job posts: local Seattle and internet Scala focused.

• College recruiting events. UW and other top CS schools.

Recruiting: Attracting Developers• Strong support from the Whitepages CTO and CEO and other management

for Scala and other advanced technologies.

• We have a great office (recently completely redone)

• We provide developers with the best workstations (top-end MacBook pros).

• We have competitive benefits, excellent work-life balance and a

collaborative supportive culture.

Training: External Resources

• Books. Odersky: Programming in Scala, and others

• Coursera. Functional programming, reactive programming.

• Typesafe training classes. Both Seattle public and in-house.

• Typesafe developer support contract and consulting. Particularly

valuable if you don’t have in-house Scala expertise.

• Conferences. We sent people to Scala Days in NY and Berlin.

Training: Internal Resources

• Whitepages developers are in a single office and come in most

days.

• This permits a high degree of personal interactions.

• Those interactions enable rapid learning and high code quality.

• This would be much harder with multiple offices and/or widely

distributed teams.

Training: Internal Approaches• Expert mentoring.

• Provide quick answers to questions.

• Detailed reviews and feedback for Ruby developers initial Scala code.

• Code reviews

• Improve code quality.

• Help those learning Scala.

• Develop a shared culture and style.

• We use a mixture of group and pair code reviews.

• Electronic means like code review tools and pull requests were less

effective.

Training: Success and Failure

• Not everyone made a successful transition from Ruby to Scala.

• Next slide give some positive and negative indicators…

Training: What to Look For

•Desire to learn new technologies.

•Desire to learn new ways to think.

•Prior functional language experience.

•Likes to work closely with others.

Scala Features to Avoid

• Scala is a very large language.

• New developers can easily get lost on the rich feature set. Often

there are several different ways of doing something, and they

don’t always know how to choose.

– We also provide the following list of features for new

developers to avoid (or to fully understand before using).

– We encourage asking more experienced developers to provide

that understanding.

– These features are also flags for discussion in code reviews.

•Reflection and macros

•asInstanceOf

•Top level objects with

mutable state

•Lazy vals

•Scala enumerations

•? (the ask pattern)

•onComplete (unless body

enclosed in try)

•Nulls

•Return, break, continue

•Mutable collections

•var, while, until

•Generics

•Implicits

•Actors

•Await

Scala Features to Avoid

We Are Hiring*!*Smart people (not just technologies) are key to our success

Thank you!Questions?