2017年のLINEのマイクロサービスを⽀えるSpringMasahiro Ide, LINE Corporation

@imasahiro

https://github.com/imasahiro

Self Introl Masahiro Idel Server side engineer at LINE

l Mainly worked on スタンプ/着せかえ ショップl Contribute to Armeria - Asynchronous RPC/REST library

l https://github.com/line/armeria

• Where is Spring used at LINE?• Useful/Customized Spring features• Monitoring

Agenda

LINE

LINE System

l LINE has hundreds of applicationsl Recent apps are Java/Spring applications

l Mainly communicates with Thrift, and REST

l Uses same in-house deployment, monitoring system

今⽇お話しするLINEのシステムlスタンプショップ

スタンプショップl Spring-boot 1.5.8, Spring-MVCl Mostly Asynchronous Http/2 Thrift/REST service

l Java8, RxJava2, (Completable|Listenable)Futurel Storage

l Redis, MySQL, Mongo, ElasticSearchl Monitoring

l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

Sticker server Mongo

MySQL

ElasticSearch

Redis

LINE Client

Theme server Search server

STORE Server

スタンプショップ 構成

スタンプショップl Spring-boot 1.5.8, Spring-MVCl Mostly Asynchronous Http/2 Thrift/REST service

l Java8, RxJava2, (Completable|Listenable)Futurel Storage

l Redis, MySQL, Mongo, ElasticSearchl Monitoring

l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

l Easy to setup

l Have many integrations with other librariesl Mybatis, Thymeleaf, Tomcat, etc.

Why spring, why spring-boot?

@SpringBootApplicationpublic class SampleApp {@Configurationclass MvcConf extends WebMvcConfigurerAdapter {…}public static void main(String[] args) {SpringApplication.run(SampleApp.class, args);

}}

Simple & Useful Spring features

1. Spring cache abstraction

1. (Spring) AOP

2. Dependency management

1. Spring Cache AbstractionlSpringにおけるCacheの抽象化

lCan choose cache implementationl Redis, Caffeine, Guava, etc.

l…But need to use carefully on async service

@Cacheable(cacheNames = "listCache")List<Item> selectByLang(String lang) {return sqlSession.selectList(…);

}

Async support in cache abstractionlNaive (and wrong) solution

l This just caches a Future…l But donʼt cache value in Futurel See https://jira.spring.io/browse/SPR-12967

@Cacheable(cacheNames = "listCache")CompletableFuture<List<Item>> selectByLang(String lang) {return CompletableFuture.supplyAsync(

() -> sqlSession.selectList(…), executor);}

Async support in cache abstractionlMade similar cache abstraction

l with Caffeine AsyncLoadingCache andLettuce (Async redis client)

@AsyncCacheable(cacheNames = "listCache")CompletableFuture<List<Item>> selectByLang(String lang) {return CompletableFuture.supplyAsync(

() -> sqlSession.selectList(…), executor);}

Simple & Useful Spring features

1. Spring cache abstraction

1. (Spring) AOP

2. Dependency management

2. (Spring) AOPlAOP - Aspect Oriented Programming

l Use case: logging, perf instrumentation@Loggable @GetMapping(“/”)ModelAndView home(…) {…}

@Around("@annotation(Loggable)")public Object logging(ProceedingJoinPoint pjp) {

Timer timer = newTimer()try {

return pjp.proceed();} finally {

long latency = timer.stop();logger.info(“request path: {}, args:{}, latency:{}”, …, latency));

}}

Client side MySQL sharding using AOP

ApplicationApplication

MySQL Slave

MySQL Master

Shard01

MySQL Slave

MySQL Master

Shard01

MySQL Slave

MySQL Master

Shard01

@ShardableList<Item> selectListByUserId(int id) {…}

List<DataSource> dataSources = …;

@Around("@annotation(Shardable)")Object sharding(ProceedingJoinPoint pjp) {

int id = getArguments(pjp)[0];source = dataSrouces.get(id % 12);useDataSource(source);return pjp.proceed();

}

3. Dependency managementlBOM and Dependency management pluginlSpringがlibrary versionを管理してくれる

dependencyManagement { imports {

mavenBom 'io.spring.platform:platform-bom:Brussels-SR5' } dependencies {

dependencySet(group: 'com.linecorp.armeria', version: '0.55.0') { entry 'armeria-tomcat-shaded’ entry 'armeria-thrift0.9-shaded'

}dependency 'biz.paluch.redis:lettuce:4.4.1.Final’…

}}

スタンプショップl Spring-boot 1.5.8, Spring-MVCl Mostly Asynchronous Http/2 Thrift/REST service

l Java8, RxJava2, (Completable|Listenable)Futurel Storage

l Redis, MySQL, Mongo, ElasticSearchl Monitoring

l Micrometer, Dropwizard Metrics, Prometheus, Zipkin

How to make a service asynchronous?l元々はTomcat+Spring MVC or thrift+DBなシステム

@Repository public class ShopStorage {List<Item> selectByLang(String lang) {

return sqlSession.selectList(“shop.selectByLang”, lang);

}}

@Controller public class ShopController {@ResponseBody List<Item> home() {

return storage.getList();}}

How to make a service asynchronous?lSync accessを１つずつAsyncに書き直す

@Repository public class ShopStorage {List<Item> selectByLang(String lang) {

return sqlSession.selectList(“shop.selectByLang”, lang);

}}

@Controller public class ShopController {@ResponseBody List<Item> home() {

return storage.getList();}}

@Repository public class AsyncShopStorage {CompletableFuture<List<Item>>

selectByLang(String lang) {return CompletableFuture.supplyAsync(() -> sqlSession.selectList(

“shop.selectByLang”, lang),executor);

}}

@Controller public class AsyncShopController {@ResponseBodyDeferredResult<List<Item>> home() {

result = new DeferredResult<>();storage.getList()

.thenApply(result::setResult);return result;

}}

• Where is Spring used at LINE?• Useful/Customized Spring features• Monitoring

Agenda

How to observe our services?lLog

l その瞬間に何が起こったかを確認するlMetric

l 現在起きている現象のトレンドを確認するlTrace

l システム間で何が何が起きているかを確認する

How to observe our services?lLog

l Slf4j + KibanalMetric

l Micormeter, Prometheus, Dropwizard metricsl Grafana

lTracel Zipkin

l io.zipkin.brave:brave-instrumentation-mysqll io.zipkin.brave:brave-instrumentation-spring-webmvcl And elasticsearch, mongo, redis integrations, etc.

Micrometerl Supports multiple monitoring tools

l Dropwizard Metrics, Prometheus, etc.

l Will be provide as a Spring Boot 2.0 metrics collection

l Already used in production serversl Found some concurrency issues but now looks stablel https://github.com/micrometer-metrics/micrometer/issues/139l https://github.com/micrometer-metrics/micrometer/issues/172l https://github.com/micrometer-metrics/micrometer/issues/208

PrometheuslSimple TSDB, Pull型lSupports querying, alerting

Application

# HELP armeria_client_response_duration_seconds# TYPE armeria_client_response_duration_seconds summaryarmeria_client_response_duration_seconds{method="sendMessage",service=”Service",quantile="0.5",} 8.194E-4armeria_client_response_duration_seconds{method="sendMessage",service=”Service",quantile="0.75",} 9.494E-4armeria_client_response_duration_seconds{method="sendMessage",service=”Service",quantile="0.90",} 9.994E-4…

/internal/metrics

Node exporterPrometheuspull ApplicationNode exporter

Micrometer+Prometheus+Grafana

Zipkin

Put all together

lArmeria - Spring, micrometer, zipkin integration@BeanThriftServiceRegistrationBean service(MyService.AsyncIface myService,

Tracing tracing) {return new ThriftServiceRegistrationBean()

.setPath("/thrift/")

.setService(THttpService.of(shopCapabilityService)

.decorate(LoggingService.newDecorator())

.decorate(MetricCollectingService.newDecorator(MeterIdFunction.ofDefault("LineShopService")))

.decorate(HttpTracingService.newDecorator(tracing))));}

Thank you