elasticsearch, logstash, and other data - logstash, and other data contents preamble and...

Download Elasticsearch, Logstash, and Other Data - Logstash, and Other Data Contents Preamble and Introduction 3 The ELK Stack 9 Elasticsearch 13 Installation and Conï¬guration 18 Command,

If you can't read please download the document

Post on 12-Feb-2018

221 views

Category:

Documents

1 download

Embed Size (px)

TRANSCRIPT

  • Elasticsearch, Logstash, and Other Data

    John Sellens

    jsellens@syonex.com

    Cascadia IT, 2015

    March 13, 2015

    Notes PDF on USB or at http://www.syonex.com/notes/

  • Elasticsearch, Logstash, and Other Data

    Contents

    Preamble and Introduction 3

    The ELK Stack 9

    Elasticsearch 13

    Installation and Configuration 18

    Command, Control, Management 32

    Monitoring and Management 44

    Logstash 49

    Installation and Configuration 53

    c20142015 John Sellens Cascadia IT, 2015 1

    Elasticsearch, Logstash, and Other Data

    Add Ons for Logstash 77

    Monitoring and Management 82

    Nginx Front End 84

    Wrap Up 89

    c20142015 John Sellens Cascadia IT, 2015 2

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    Preamble and Introduction

    c20142015 John Sellens Cascadia IT, 2015 3

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    Overview

    Elasticsearch a search engine

    Logstash filters inputs to outputs

    Kibana web interface to Logstash/Elastisearch

    The ELK Stack

    Introduction, Installation, Configuration

    How to get things in and out

    c20142015 John Sellens Cascadia IT, 2015 4

    Notes:

    Both Cascadia IT and I will very much appreciate your feedback

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    Outline/Timetable

    Preamble / Introduction / Outline

    Elasticsearch

    Overview and concepts

    Installation, configuration, care and feeding

    Logstash

    Overview and how it fits together

    Installation, configuration, management

    The ins and outs

    Kibana and its use

    Break 3:30 to 4:00, Wrap up 4:50pm

    c20142015 John Sellens Cascadia IT, 2015 5

    Notes:

    Scheduled for 1:30 - 4:50pm with one half hour break

    Im hoping the timing fits together as I hope it will

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    Questions?

    Got a Question?

    A Clarification?

    Some Confusion?

    A Point of Interest?

    Ask!

    c20142015 John Sellens Cascadia IT, 2015 6

    Notes:

    This slide is here to be even more explicit that questions and comments

    are more than welcome, and that interactivity is good.

    Get my attention through any appropriate means, but if youre throwing

    something, please lob, and keep it light.

    Though please consider the time we have available before you start on a

    long, involved anecdote of what once happened to a friend of yours.

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    About the Instructor

    John Sellens

    25+ years as UNIX system administrator

    University of Waterloo, UUNET, managed services,

    FreshBooks, NightingaleMD . . .

    Long time USENIX and LISA attendee and speaker

    And elsewhere too . . .

    Occasional writer and author

    c20142015 John Sellens Cascadia IT, 2015 7

    Notes:

    Feel free to contact me here or by email if you have any questions

  • Elasticsearch, Logstash, and Other Data Preamble and Introduction

    Viewpoints and Religion

    I like simple

    And like making my job easier, not harder

    Multiple cooperating component parts are good

    AKA The UNIX Philosophy

    Not too crazy about the bleeding edge

    Solve any problem in computer science with another level of

    indirection

    But not too much of that today

    c20142015 John Sellens Cascadia IT, 2015 8

    Notes:

    With that viewpoint, generally I think Elasticsearch and friends are pretty

    cool

    Im not generally a Java fanatic or anything like that, but these tools seem

    well implemented

  • Elasticsearch, Logstash, and Other Data The ELK Stack

    The ELK Stack

    c20142015 John Sellens Cascadia IT, 2015 9

  • Elasticsearch, Logstash, and Other Data The ELK Stack

    The ELK Stack - Elasticsearch Logstash Kibana

    No master plan to take over the world?

    General need for scalable text search

    Elasticsearch built on Lucene

    Nice distributed, reliable database

    Hey! That might be a good place to collect log files!

    How about a convenient way to query the log data?

    Seems to have gained prominence fairly quickly

    c20142015 John Sellens Cascadia IT, 2015 10

    Notes:

    Or at least this is my impression of how things might have happened

    I could be full of nonsense of course

  • Elasticsearch, Logstash, and Other Data The ELK Stack

    The ELK Ecosystem

    Primary components are open-source

    Developers formed Elasticsearch the company

    Services, support, add-ons

    Upcoming more enterprisey tools

    Logstash and Kibana joined in

    Lots of people doing tools, docs, blogs, . . .

    Starting to generate other products

    e.g. Nagios Log Server

    c20142015 John Sellens Cascadia IT, 2015 11

    Notes:

    elaseticsearch.org

    elaseticsearch.com

    Formed in 2012, seems well-funded

    Simple is best

    Seems like a healthy environment to hitch your wagon to

    Standard docs and repositories for RPM-ish and APT-ish systems

  • Elasticsearch, Logstash, and Other Data The ELK Stack

    One or Many

    ELK can be self-contained, on a single machine

    As we shall see, lapotp willing . . .

    Most components can be split to multiple machines

    Elasticsearch clusters

    Logstash shippers, brokers, indexers

    Some useful related parts are missing

    e.g. Security and access controls

    c20142015 John Sellens Cascadia IT, 2015 12

    Notes:

    The standard packages install and start with a usable configuration

    Most people will want to do something more advanced than a single ma-

    chine

    But you can easily put a demo system together

    Can be installed and configured with configuration management tools

    Puppet and the like

    Typically we run ELK on UNIX-ish servers

    But its java, so it can run anywhere, right?

    I think logstash can run on Windows, and grab from the eventlog

    Though using something like nxlog on Windows might make more

    sense

  • Elasticsearch, Logstash, and Other Data Elasticsearch

    Elasticsearch

    c20142015 John Sellens Cascadia IT, 2015 13

  • Elasticsearch, Logstash, and Other Data Elasticsearch

    What is Elasticsearch?

    Elasticsearch is a search engine

    Distributed, scalable, resilient, HA

    RESTful API, JSON, HTTP

    Built on Apache Lucene

    Stores documents

    Organized by type

    In an index

    c20142015 John Sellens Cascadia IT, 2015 14

    Notes:

    The reference docs are

    http://elasticsearch.org/guide/en/elasticsearch/reference/current

    And worth a read (or a perusal, or . . . )

    And the glossary

    http://elasticsearch.org/guide/en/elasticsearch/reference/current/glossary.html

  • Elasticsearch, Logstash, and Other Data Elasticsearch

    Documents, Types, Indexes

    Documents are JSON documents

    So they can have some structure to them

    In RDBMS terms:

    Index database

    Type table

    Document row

    Documents have a document id

    indexname/type/id is the unique identifier

    Documents can have version, TTL, parent/child

    c20142015 John Sellens Cascadia IT, 2015 15

    Notes:

    JSON list of keyword : value pairs, plus more!

    More information about document attributes is in the documentation for

    the index API

  • Elasticsearch, Logstash, and Other Data Elasticsearch

    How About That Index?

    An index can be created implicitly or explicitly

    i.e. You can just start shoving documents in

    An index is divided into shards

    Each shard is a lucene instance

    And may have replicas of the shards

    Replicas for reliability and read performance

    Mappings give hints about data types of fields

    To help make indexing and searching more efficient

    Aliases are like database views

    Combine multiple indexes, select with a filter

    c20142015 John Sellens Cascadia IT, 2015 16

    Notes:

    The index API adds or updates a document to an index, and will automat-

    ically create an index when first used

    The create index API lets you explictly create an index and set its at-

    tributes

    Mappings will be created automatically if not specified

    A mapping is like a schema definition in a relational database

    More later on deciding on the number of shards and replicas

  • Elasticsearch, Logstash, and Other Data Elasticsearch

    All About the API, Not the CLI

    At first, I was confused

    Where are the administration commands?

    Everything is a RESTful API call

    i.e. Its all through HTTP interaction

    Much admin-type stuff is done with curl or similar

    Lack of access controls make this much simpler

    Which may or may not be a feature

    curl http://es01:9200/curl http://es01:9200/idxname/_status?pretty

    c20142015 John Sellens Cascadia IT, 2015 17

    Notes:

    Hopefully, Im less confused now

  • Elasticsearch, Logstash, and Other Data Installation and Configuration

    Installation and Configuration

    c20142015 John Sellens Cascadia IT, 2015 18

  • Elasticsearch, Logstash, and Other Data Installation and Configuration

    Planning Ahead

    Its worthwhile to consider your environment

    Will affect networking, configuration, shards, replicas . . .

    If a single machine will do it all, its

Recommended

View more >