![Page 1: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/1.jpg)
Serverless Real-Time Analysis
Minyoung JeongCTO, The Beatpacking Company
![Page 2: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/2.jpg)
Minyoung Jeong
CTO, The Beatpacking Company
Founder, AWSKRUG
AWS Community Hero
Love Python!
![Page 3: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/3.jpg)
Agenda
• Real-Time Analysis ?
• Why Serverless?
• Architecture Overview
• AWS Products
![Page 4: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/4.jpg)
Real-Time Analysis
![Page 5: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/5.jpg)
BEATAd supported Streaming Radio
![Page 6: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/6.jpg)
To sell ads,need Real-time analysis
![Page 7: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/7.jpg)
To detect abusing, use Real-time analysis.
![Page 8: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/8.jpg)
What's Real-time?
![Page 9: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/9.jpg)
Differences between Unit
![Page 10: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/10.jpg)
Day, Hour, Minute
![Page 11: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/11.jpg)
Day, Hour, Minute
![Page 12: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/12.jpg)
Second
![Page 13: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/13.jpg)
충분히 적은 오차를 감수지연을 줄이는 분석
![Page 14: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/14.jpg)
Why Serverless?
![Page 15: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/15.jpg)
–William of Occam
“Pluralitas non est ponenda sine neccesitate”
![Page 16: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/16.jpg)
Everything seems nails when you hold a hammer?
![Page 17: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/17.jpg)
Flexible by demand Increase efficiency
![Page 18: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/18.jpg)
Deploy, Scaling, CapacityFault tolerance,, ….
![Page 19: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/19.jpg)
Works great with Microservice Architecture
![Page 20: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/20.jpg)
Service that covers smaller areaQuick develop, higher reliability
![Page 21: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/21.jpg)
MonitoringLoad dispersion and management
![Page 22: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/22.jpg)
Forget everything other than codes.
![Page 23: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/23.jpg)
Forget everything other than codes ?
![Page 24: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/24.jpg)
Lambda has inconvenient development model Apex, Serverless
![Page 25: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/25.jpg)
Architecture Overview
![Page 26: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/26.jpg)
![Page 27: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/27.jpg)
![Page 28: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/28.jpg)
![Page 29: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/29.jpg)
![Page 30: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/30.jpg)
![Page 31: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/31.jpg)
![Page 32: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/32.jpg)
![Page 33: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/33.jpg)
Peak: 1K RPS Avg: 300 RPS
![Page 34: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/34.jpg)
API Gateawy+Lambda+ES<= $400/m < m4.2xlarge 1ea/m
![Page 35: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/35.jpg)
API Gateway
![Page 36: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/36.jpg)
REST EndpointAuth & Cache & Monitoring
![Page 37: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/37.jpg)
HTTP ProxyLambda, AWS Service
![Page 38: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/38.jpg)
![Page 39: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/39.jpg)
![Page 40: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/40.jpg)
![Page 41: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/41.jpg)
By using Cloufront,reduce latency and prevent DDOS
![Page 42: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/42.jpg)
Automatically expands within designated Range, then throttling works.
![Page 43: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/43.jpg)
Utilizing Stage feature,Test/Verification
![Page 44: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/44.jpg)
![Page 45: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/45.jpg)
Request/Response Template
![Page 46: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/46.jpg)
Kinesis
![Page 47: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/47.jpg)
Platform for streaming data on AWS
![Page 48: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/48.jpg)
Managed KafkaQueue/ETL/Analysis
![Page 49: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/49.jpg)
Scale In/Out by ShardExplore forward, backward, and certain point of streamConfigurable Retention Period
![Page 50: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/50.jpg)
Stream
Shard
Shard
Shard
Shard
![Page 51: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/51.jpg)
TRIM_HORIZON LATESTAT(AFTER)_SEQUENCE
![Page 52: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/52.jpg)
Connect various consumers to one streamInvocation is available within retaining period Idempotence
![Page 53: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/53.jpg)
Lambda Integration(w/Event Source)Provide KCL/KPL SDKREST API
![Page 54: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/54.jpg)
Lambda
![Page 55: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/55.jpg)
Code in the Cloud
![Page 56: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/56.jpg)
Execute by receiving event and contextCRON/AWS Event/API GatewayCloudwatch Log & Monitoring
![Page 57: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/57.jpg)
![Page 58: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/58.jpg)
Set only memory and execution timeoutAutoscale by throughputCharge by 100ms
![Page 59: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/59.jpg)
Throtting and latency spike existExecute invocation and assure idempotence
![Page 60: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/60.jpg)
event
![Page 61: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/61.jpg)
function endpointCollect(event, context) { // api-gateway context let apiContext = event.context;
event.body._context = { es_name: apiContext['es-name'], es_index: apiContext['es-index'], es_doc_type: apiContext['es-doc-type'], s3_bucket: apiContext['s3-bucket'], s3_prefix: apiContext['s3-prefix'] }
putStream(apiContext['stream-name'], event).then((resolve, reject) => { context.succeed(); }).catch((err) => { context.fail(err); }); }
![Page 62: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/62.jpg)
function putStream(streamName, event) { // inbound from AWS API Gateway
let injected = injectContext(event); let record = { Data: JSON.stringify(injected), PartitionKey: `${event.body.timestamp}`, StreamName: streamName }; return new Promise((resolve, reject) => { kinesis.putRecord(record, (err, data) => { if (err) { reject(err); } else { resolve(data); } }); }); }
![Page 63: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/63.jpg)
const ESEndpoint = new AWS.Endpoint(es_host); const req = new AWS.HttpRequest(ESEndpoint);
req.method = 'POST'; req.body = body; req.path = '/_bulk'; req.region = 'ap-northeast-1';
req.headers['presigned-expires'] = false; req.headers['Host'] = ESEndpoint.host;
if (AWSCredential.accessKeyId) { const signer = new AWS.Signers.V4(req, 'es'); signer.addAuthorization(AWSCredential, new Date()); }
![Page 64: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/64.jpg)
Demo
![Page 65: [AWSKRUG&JAWS-UG Meetup #1] Serverless Real-Time Analysis](https://reader034.vdocuments.net/reader034/viewer/2022052706/5882d0de1a28abf8388b5d45/html5/thumbnails/65.jpg)
Thanks!