exploring twitter's finagle technology stack for microservices
TRANSCRIPT
Exploring Twitter’s Finagle technology
stack for microservicesTomasz Kogut / @almendar
XII Tricity Scala User Group Meetup
About me:• Software Engineer at
• Working in the internet advertisement industry
• I try to create distributed, highly performant systems that will run on the JVM
• Programming in Scala since ~2011
Microservices
questionnaireWho uses μ-services approach?
Who has a system that would like to break down into smaller pieces?
What we already know?
• Working on huge monolithic projects is hard
• Complexity and interdependencies become incomprehensible at some point
• We all fell in our guts that breaking it down is a way to go
• Single responsibility principle but for services
The plan1. Create multiple git repositories
2. Create/copy/move code around repos
3. Pour it all with a fair share of REST/HTTP
4. Extract common code/models into libs
5. Run it!
Let’s go to production!
So what happened?
Paying for the lunch
• Easy to develop
• Hard to operate
• More fine-grained services boost both traits
Things to address
• A lot of stuff from “Fallacies of distributed computing"
• Tracing request across different services
• Service discovery and tracing service topology
• Message serialization
• Distributing load across nodes
• Measuring performance and finding bottlenecks is not easy
• Interface contracts can introduce hidden coupling
• Duplication of effort
• Historically we were a .Net shop
• Think spring-like approach
• Moved to API first approach (although we were running some kind of μ-service approach to some extend)
• Splitting one of the business features into small-chunks using whatever tech stack we want so we went akka/akka-http
• We didn’t want to use akka clustering
• Recreating utils e.g. request retry
• Establishing new project = work upfront before creating any business value e.g. metrics
• Every project had it’s own structure - mental context switching
• REST is good but other protocols might be better
• Fighting the “spray DSL” and marshallers 😀
• Creating json is annoying and automatic derivation from classes was useless most of the time
• We had to think how to properly version our apis and hold backward compatibility
• We had situations where multiple services needed to be deployed for the solution to work (hidden monolith)
• How do I run this service locally?
What is it?• Protocol agnostic RPC system for JVM
• Based on Netty
• Allows for building servers and clients
• Core platform on which Twitter is build
• "Your Server as a Function" by Marius Eriksen
Foundations• Futures• Services• Filters
Future• NOT the Scala one unfortunately but very close
• Some api differences- import com.twitter.util.Future- Future.value == Future.successful- Future.exception == Future.failed- Future.collect == Future.sequence
• Usual monadic composition with flatMap and map
• There is also twitter’s Try, Duration and a bunch of others…
Service• Basic building block in Finagle
• Just a function that returns a Future
• Servers implement services to which Finagle dispatches incoming requests
• Finagle furnishes clients with instances of Service representing either virtual or concrete remote servers
abstract class Service[-Req, +Rep] extends (Req => Future[Rep])
Example local service (1)
import com.twitter.finagle.Serviceimport com.twitter.util.Future
val lengthService = Service.mk[String, Int] { req =>
Future.value(req.length)
}
lengthService("Hello TSUG").foreach(println)
import com.twitter.finagle.httpimport com.twitter.finagle.Serviceimport com.twitter.util.Futureimport com.twitter.finagle.Http
val service = new Service[http.Request, http.Response] { def apply(req: http.Request): Future[http.Response] = Future.value( http.Response(req.version, http.Status.Ok) )}val server = Http.serve(":8080", service)
Example http server (2)
import com.twitter.finagle.http.Method.Getimport com.twitter.finagle.{Http, Service}import com.twitter.finagle.http.{Request, Response}import com.twitter.finagle.service.FailFastFactory.FailFastimport com.twitter.util.Future
val productServiceHttpClient: Service[Request, Response] = Http.client.configured(FailFast(false)).newClient("127.0.0.1:8080", "productService").toService
val response: Future[Response] = productServiceHttpClient(Request(Get,"/products/22"))
response.foreach{ response => println(response.statusCode) println(response.contentString)}
Example http client (3)
Other protocols client examples
var memCache = ClientBuilder() .name("mc") .codec(Memcached()) .hostConnectionLimit(config.concurrency()) .hosts(config.hosts())
val mySqlClient = Mysql.client .withCredentials(username(), password()) .withDatabase(dbname()) .newRichClient("%s:%d".format(host().getHostName, host().getPort))
Services represents clients and servers
symmetrically
Please notice that:
Filter• Again a function that returns a Future
• Filters are always attached to Services altering their behavior
• Used for application agnostic concerns such as: timeouts, retry policies, service statistics, and authentication
abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])
Filter• Filters just like any other functions
compose
• There is a andThen method that chains different filters with each other and in the end with service
Local service filter
import com.twitter.finagle.{Filter, Service}import com.twitter.util.Future
val someStringMetrics = Service.mk[String,Int] { req => Future.value(req.length)}
val evenMetricsFilter = new Filter[Float, Boolean, String, Int] { override def apply(input: Float, stringMetricsService: Service[String, Int]): Future[Boolean] = { stringMetricsService(input.toString).map(_ % 2 == 0) }}
(evenMetricsFilter andThen someStringMetrics)(1.234f)
import com.twitter.finagle.{Http,Service, Filter}import com.twitter.finagle.http.{Request,Response}import com.twitter.finagle.service.TimeoutFilterimport com.twitter.util.MockTimerimport com.twitter.conversions.time._
val httpClient: Service[Request, Response] = Http.client.newService("twitter.com")val timeoutFilter: Filter[Request, Response, Request, Response] = new TimeoutFilter[Request, Response](30.seconds, new MockTimer)val httpClientWithTimeout: Service[Request, Response] = timeoutFilter andThen httpClient
Filter example http client
WARNINGThis is only a showcase, timeouts for http clients are best configured by Http.client object
How to implement this?
Stacking filters
recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer
Fronted webservers configuration at twitter:
Cool client filters (or wannabe-filters)
• MonitorFilter - handling exceptions
• StatsFilter - exposing metrics
• RetryFilter - retying calls with configured budget and a back-off policy
• TimeoutFilter - mentioned earlier, for different aspects like idle time, request time, connection time ect.
• LoadBalancing (not really a filter) - distributed load across nodes, just pass multiple host values
• FailFast (not really a filter) - mark node as dead for a certain period of time
• Unfortunately not all functionality can be expressed with Filters
• Some things are a bit hairy - e.g. request cancelation
• Beneath implementations is non-trivial but comprehensible
• But the good part is that it all works out to the box for free with almost none configuration!
There is more in Finagle to explore:
• Thrift protocol
• Mux - multiplexing RPC
• ZooKeeper support
• Network location naming
Twitter Server
Even more goodies
Twitter Server• A “template” for finagle-based
server
• Flags
• Logging
• Metrics
• Admin interface
Flags (1)• Cmd line arguments passed to the
application
• A remedy to “How do I start this thing and what are the options?”
• Smart with type inference and parsing capabilities
Flags (2)val addr = flag("bind", new InetSocketAddress(0), "Bind address")val durations = flag("alarms", (1.second, 5.second), "2 alarm durations")
$ java -jar target/myserver-1.0.0-SNAPSHOT.jar -helpAdvancedServer
-alarm_durations='1.seconds,5.seconds': 2 alarm durations -help='false': Show this help
-admin.port=':9990': Admin http server port -bind=':0': Network interface to use
-log.level='INFO': Log level -log.output='/dev/stderr': Output file
-what='hello': String to return
Option for fail fast when missing a flag
Logging• Twitter server provides a logging
trait
• It can be configured with standard flags
• Can be tuned on-the-fly
Admin interface loggers control
Metrics• Defining own statistics
• JVM stats
• Processing and network stats
Others• Linting your server for problems
• Looking at threads
• Profiling
Admin interface
Finatra
What is it?• Finatra is build atop Finagle and
Twitter Server
• Finatra does all the heavy-lifting that one needs to do when working only with Finagle
Finatra features• Dependency injection using Guice
• JSON handling with Jackson
• Mustache support
• Routing
• Integrates it all together nicely
package pl.tk.finagle.recommendation.controller
import javax.inject.{Inject, Singleton}
import com.twitter.finagle.http.Requestimport com.twitter.finagle.tracing.ClientTracingFilter.TracingFilterimport com.twitter.finatra.http.Controllerimport com.twitter.finatra.request._import com.twitter.inject.Loggingimport pl.tk.finagle.recommendation.engine.{RecommendationCmd, RecommendationEngine}
case class RecommendationFromCookieRequest(@Inject request: Request, @RouteParam cookieId: Int, @RouteParam eshopId: Int, @Header `x-auth`: String)
@Singletonclass RecommendationController @Inject()(recommendationEngine: RecommendationEngine) extends Controller with Logging {
get("/recommend/:eshop_id/:cookie_id") { r: RecommendationFromCookieRequest => infoResult("Hello") { TracingFilter("RecommendationEngine") andThen recommendationEngine apply RecommendationCmd(r.cookieId, r.eshopId) } }}
Unified error reporting
➜ finagle-spike git:(master) ✗ curl -v 127.0.0.1:2001/recommend/32/wsf* Trying 127.0.0.1...* Connected to 127.0.0.1 (127.0.0.1) port 2001 (#0)> GET /recommend/32/wsf HTTP/1.1> Host: 127.0.0.1:2001> User-Agent: curl/7.43.0> Accept: */*>< HTTP/1.1 400 Bad Request< Content-Type: application/json; charset=utf-8< Server: Finatra< Date: Wed, 30 Mar 2016 07:05:53 +00:00< Content-Length: 79<* Connection #0 to host 127.0.0.1 left intact{"errors":["cookie_id: 'wsf' is not a valid int","x-auth: header is required"]}%
JSON Support• Build on jackson-module-scala
• Support for case classes (de)serlialization
• Integrated with Joda
• Error accumulation (instead of fail-fast)
• Integration with routing
• Most of the cases just return object
Customizing Json
class Server extends HttpServer { override def jacksonModule = CustomJacksonModule ...}
object CustomJacksonModule extends FinatraJacksonModule { override val additionalJacksonModules = Seq( new SimpleModule { addSerializer(LocalDateParser) })
override val serializationInclusion = Include.NON_EMPTY
override val propertyNamingStrategy = CamelCasePropertyNamingStrategy
override def additionalMapperConfiguration(mapper: ObjectMapper) { mapper.configure(Feature.WRITE_NUMBERS_AS_STRINGS, true) }}
Validation
import com.twitter.finatra.validation._import org.joda.time.DateTimecase class GroupRequest(@NotEmpty name: String, description: Option[String], tweetIds: Set[Long], dates: Dates) { @MethodValidation def validateName = { ValidationResult.validate( name.startsWith("grp-"), "name must start with 'grp-'") }}
case class Dates(@PastTime start: DateTime, @PastTime end: DateTime)
Also checked at routing time
Testing (1)• Based on ScalaTest
• Feature testing (both black box and white box) - looking at external interface of our service
• Integration tests - only a subset of modules is instantiated and tested
• Finatra does not provide anything for unit testing
Testing (2)• Smart JSON diff
• Integration with DI
• Easy mocking
• Embedded http server with our service
last but not least…
What is it?• Distributed request tracing
• Based on Google Dapper paper
• Helps getting insight on how the services interact
Zipkin
Python pyramid_zipkinPyramid Http
(B3)Http (B3) Kafka | Scribe Yes
py2, py3
support.
Java brave
Jersey,
RestEASY,
JAXRS2,
Apache
HttpClient,
Mysql
Http (B3)Http, Kafka,
ScribeYes
Java 6 or
higher
Scala finagle-zipkin FinagleHttp (B3),
ThriftScribe Yes
Ruby zipkin-tracer Rack Http (B3)Http, Kafka,
ScribeYes
lc support.
Ruby 2.0 or
higher
C# ZipkinTracerM
odule
OWIN,
HttpHandlerHttp (B3) Http Yes
lc support.
4.5.2 or higher
Go go-zipkin
Not tied to finagle
Zipkin Architecture
DEMO APP
Recommendation
ProductsService
UserProfile
1 GetProducts
2 GetPromotedProducts
GetUser
CookieId, Eshop
Thank you!