Evolution of AWSBackend of SmartNews2016/01/27 Classmethod x SmartNews AWS Night

Self Introduction

• Nobutoshi Ogata

• Manager, Site Reliability Engineering

• @nobu666

• ❤ Whiskey, Cat, Heavy Metal

• Entrusted dev.(10y) ➡ Game platform infrastructure(3y) ➡ Some startup(1y) ➡ SmartNews(2015/05-)

SmartNews

15,000,000+downloadsworldwide

SmartNews 3.0

LINE Stampshipped today

! Channel!

! Channel!

Questionnaire

Agenda• Scale

• Services

• Changes in the configuration

• Configuration management

• Future plan

Scale

Scale• 300+ EC2 instances

• 80 ELBs

• 70 Auto Scaling Groups

• 80+ S3 buckets (300+TB)

• 40 RDS instances (MySQL & Aurora)

• 100 Kinesis shards

• 10 EMR clusters (Hive+Spark)

Scale• 50+ ElastiCache clusters (Redis + Memcached)

• 30+ CodeDeploy applications

• 70,000,000 CloudSearch seachable documents

• 8 Redshift nodes

• 10+ Lambda functions

• 80+ SQS queues

• 20+ SNS topics

Services

Changes in theconfiguration

Stone Age

AnalyzerCrawlerIndexer

On-memory DBAPI

Monolithic Application

Bronze Age

AnalyzerIndexer

On-memory DBAPI

Crawler

Crawler

Crawler

Queue

Split Crawler

Iron Age

API Indexer Analyzer Crawler

Split more

Present day

Crawler

Nationality detector

Language detector

Analyzer

Facial recognition

Indexer

Ranking

Search

Recommend

API Gateway

Admin tool Validator

Proxy

Live

Push

Sexy detector

Auth Proxy

Micro Services?

• SmartNews's journey into microservices

• インフラ専任エンジニアが一人もいないSmartNewsにおけるクラウド活用法

• SmartNews Ads

• Next session about data platform

Configurationmanagement

Pull style deploy

➡ ➡ ⬅ • CodeDeploy was used to automate deployment tasks

• Operating an AWS ELB

• Fetching a new revision

• Restarting services with our custom program

• e.g. downloading machine learning model data, precompiling assets and deploying Docker containers

Auto Scaling Group• There is an advantage even if you don't want to autoscaling

• Easy to dispose

• replace kernel, switch to other OS, ...

• Easy A/B testing

• change config only specific server

• tag:Attributes

• Instance Protection

ASG ! InstanceName• We were stop using InstanceName

• Using tag:Group and tag:Env

Infrastructure as Code• Route53 ➡ winebarrel/roadworker

• Security Group ➡ winebarrel/piculet

• Elastic IP Addresses ➡ winebarrel/eipmap

• Provisioning ➡ itamae-kitchen/itamae

• Datadog ➡ airbnb/interferon

DSL managementis not perfect• Complicated automation

• ChatOps ➡ change code ➡ git add && git commit && git push

• Too much hassle... !

Run Command• Secure

• IAM, CloudTrail

• Easy CI integration w/o SSH

• be able to use EMR also

Exampleaws ssm send-command \--document-name "AWS-RunShellScript" \--instance-ids \ $(aws emr list-instances \ --cluster-id \ $(aws emr list-clusters --active | \ jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') | \ jq -r '.Instances[].Ec2InstanceId') \--parameters '{"commands":["ls"],"executionTimeout":["10"]}' \--timeout-seconds 10 \--output-s3-bucket-name "ssm-logs" \--output-s3-key-prefix "prefix-" \--region ap-northeast-1

Future plan

ECSSpot Fleet

Contaniner RegistryConsul

AWS Blackbelt 2015 ECS p.98

Containerizeall services

ScalableSafetyEfficientPortable

Measurable

Aggregates events

Events aggregator• Events

• Deploy/Provision/Restart/Terminate etc.

• All events goes to SNS or Kinesis. Users will describe and hook scripts

• Visualize via Datadog

• Notice to Slack

• Auto healing

Summary

Summary• AWS release new service continuously

• We ride on the shoulders of giants

• We will never forget that OSS helped us

We're hiring!Only two people onSite Reliability Engineering Team

http://about.smartnews.com/en/careers/

Credithttps://flic.kr/p/hAcPRk taken by Niuton mayhttps://flic.kr/p/9UnRWS taken by Alberto G.https://flic.kr/p/ydnEvw taken by ep_jhuhttps://flic.kr/p/pyfnTV taken by Kevin Dooleyhttps://flic.kr/p/mtLpc4 taken by L1meyhttps://flic.kr/p/fgG226 taken by Linux Screenshotshttps://flic.kr/p/7DsuLz taken by Gloson Tehhttps://flic.kr/p/jfGpn7 taken by julochka