login with aws elasticsearch servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf ·...
TRANSCRIPT
Login with AWS Elasticsearch Services.NE T CO R E
Overview
• Released in 2006
• Previous version not supported for structured login
• Introduced structured login in 2017 which is support for .NET standards
Overview
• The newest login framework
• Released in 2013
• The big difference between Serilog and the other frameworks is that it is designed to do structured login out of the box
• Serilog, was.. NLog also supports structured login, but it is was only added recently and the benchmarks suggest that using it has some serious performance implications.
Serilog
• Structured login support in Serilog
• Ease of configuration
• API is more modern
• Ability to add enrichers give you the ability to intercept and modify the messages (append properties to the log messages)
• Can store logs in different destinations (Sinks)
• Ex: Amozon Cloudwatch, Azure Blob Storage, Azure Analytics, Rolling File, Azure Event Hub, Elasticsearch
Serilog
NLog Benchmark
Serilog Benchmark
Conclusion
• Serilog log provider and the NLog log provider are similarly configured
• The results for Serilog are much better than the ones for NLog in both throughput and latency.
• In most cases latency for Serilog is half as much as the latency for NLog, and throughput twice as much.
Serilog
• Advantages
• Widely adopted
• A lot of documentation
• Great community
• Disadvantages
• A bit harder to learn when coming from log4net
NLog
• Advantages
• A lot of documentation
• Having been around a long time, there are lots of blog posts
• Easy to get started when coming from other login frameworks
• Disadvantages
• Structured login is still a bit behind Serilog
• C#-based API is harder to use than Serilog's fluent API
Centralized login in .Net Core using AWS Elasticsearch Service
Serilog & Kibana
• Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.
• It is No Sql, distributed, full text database
• built on Apache Lucene and was first released in 2010
• Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of open source tools for data ingestion, enrichment, storage, analysis, and visualization
• Referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana)
Elasticsearch Used For?• Application search
• Website search
• Enterprise search
• login and log analytics
• Infrastructure metrics and container monitoring
• Application performance monitoring
• Geospatial data analysis and visualization
• Security analytics
• Business analytics
How does Elasticsearch work?
• Raw data flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications
• Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch.
Elasticsearch Index
• Collection of documents that are related to each other
• Stores data as JSON documents
• Document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data).
Logstash
• Core products of the Elastic Stack
• Used to aggregate and process data and send it to Elasticsearch
• An open source, server-side data processing pipeline
• Enables you to ingest data from multiple sources simultaneously and enrich and transform it before it is indexed into Elasticsearch
Kibana
• Data visualization and management tool for Elasticsearch
• Provides real-time histograms, line graphs, pie charts, and maps
• Includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.
Why use Elasticsearch?
• Elasticsearch is fast
• Elasticsearch is distributed by nature
• Elasticsearch comes with a wide set of features
• The Elastic Stack simplifies data ingest, visualization, and reporting
Elasticsearch Logging Process
How to Setup AWS Elasticsearch Service
• Setup Elastic Search Domain
• Configure Cluster
• Setup Access
• Setup Access Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:762493928111:domain/youcab/*"
}
]
}
Elasticsearch Service Summary
.Net Core Application Setup
• Required Packages
• Elasticsearch.Net
• Elasticsearch.Net.AWS
• Serilog.Sinks.Elasticsearch
Setup AWS Elasticsearch Credentials (appsetting.config)
"AWS": {
"AccessKey": "AKIAJOUREKVWNFMYLKOA",
"SecretKey": "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",
"Region": "us-east-1",
"ElasticUrl": "https://search-youcab-paf2n6jkgms3y5kxtzajjywspm.us-east-1.es.amazonaws.com"
}
Logger Service Project Setup (.Net Core)
• Setup AWS credentials in Startup class
• Configure AWS Elasticsearch Connection String
Demonstration
Elasticsearch Available Methods
• AWS Elasticsearch
• Azure Elasticsearch
• Elasticsearch Cluster
Any Questions???