kelk stack on aws
TRANSCRIPT
SteamhausKELK ON AWS
Who am I?Sean Clerkin Senior Site Reliability Engineer
Logging is difficult
No centralised loggingUser needs OS
knowledgeDistribution
Of keys
Enemy of autoscaling
Log rotation
Users download logs unnecessarily
Doesn’t scale To many servers
Slow to find issues
Alertingis hard
Sshing to servers :(
SteamhausKELK ON AWS
ELK is awesomE
SteamhausKELK ON AWS
ELK on ec2
SteamhausKELK ON AWS
KELK on AWS
• Low maintenance - No ec2, Uses entirely AWS serverless technologies and services
• ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs
• Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics
• Automated with Terraform
• Open source
Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics
SteamhausKELK ON AWS
How does it work?
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
SteamhausKELK ON AWS
Automationcode
Sample Web Stack
VPC
ALB
EC2
LoggingStack
Kinesis
ElasticsearchService
Lambda
S3
CloudfrontPython
Terraform
Do try this at home!github.com/steamhaus/kelk-example
SteamhausKELK ON AWS
Callouts from the build• It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat
SteamhausKELK ON AWS
Any Questions..?
Thank you :)Contact [email protected] 820 2020@steamhausmcr
Locate us
Fourways House57 Hilton Street Manchester M1 2EJ