introduction to shield and kibana

Shield & Kibana

Sushil KumarSoftware ConsultantKnoldus Software LLP.

Agenda

Shield

Introduction

Installation

Basic Authentication

Role-based Access Control

Message Authentication

Kibana

Introduction

Installation

Using Kibana with Shield

Getting Started with Kibana

Demo

Shield

Shield is a plugin for Elasticsearch that enables you to easily secure an elasticsearch cluster.

With Shield, you can protect your data with username and passwod.

It also provides advance security features such as encrypting communications, role-based access control, IP filtering, and auditing.

Introduction

Alternatives:

Search Guard - Free (and open source) plugin from floragunn UG

https://github.com/salyh/elasticsearch-security-plugin https://github.com/sonian/elasticsearch-jetty https://github.com/codelibs/elasticsearch-auth

Features

Preventing unauthorized access with password protection, role-based access control, and IP filtering.

Preserving the integrity of your data with message authentication and SSL/TLS encryption.

Maintaining an audit trail so you know whos doing what to your data.

IP-based authorization. You can whitelist and blacklist specific IP addresses or subnets to control network-level access to a server.

Shield preserves the integrity of your data by encrypting communications to and from nodes and authenticating messages to verify that they have not been tampered with or corrupted in transit during node-to-node communication.

Installation

Run bin/plugin install from ES_HOME to install the license plugin

Run bin/plugin install to install the Shield plugin into Elasticsearch.

Note: The Shield plugin must be installed on every node in the cluster. If you are installing to a live cluster, you must stop all of the nodes, install Shield, and restart the nodes.

bin/plugin install licensebin/plugin install shield

Uninstalling Shield

To uninstall Shield:

Shut down Elasticsearch.

Remove the Shield plugin from Elasticsearch:

Restart Elasticsearch.

bin/plugin remove shield


Once Shield is installed, a username and password is required to communicate with the cluster.

All you need to do to use basic authentication is set up users and assign them to one of the basic predefined roles:

admin: Can perform any cluster or index action.

power_user: Can monitor the cluster and perform any index action.

user: Can perform read actions on any index.

A secure Elasticsearch cluster manages the privileges of users through roles. A role has a unique name and identifies a set of permissions that translate to privileges on resources. A user can have an arbitrary number of roles. There are two types of permissions: cluster and index.


Set up UsersUse the esusers command line tool to create an user:Example:

Now you can submit requests as your admin user:

bin/shield/esusers useradd admin -r admin -p password

curl -u admin -XGET 'http://localhost:9200/'

curl -u admin:password -XGET 'http://localhost:9200/'

A realm is a user database configured for the use of the Shield plugin.

In the native realm, users are stored and distributed with the Elasticsearch cluster. With the native realm, the administrator manages users with the API for user management.

In the file realm, users exist in files stored on each node of the Elasticsearch cluster.


Delete User

List Users

Update Roles

bin/shield/esusers userdel username

bin/shield/esusers list

bin/shield/esusers roles username -r admin -a user

The file realmis now functions as fallback/recovery realm.

For example, if all users lock themselves out of the system (no one remembers their username and password), you can define an admin user in the file realm and use those credentials to restore access..

User API

The Users API enables you to create, read, update, and delete native users from the native realm.

To use this API, you must have at least the manage_security cluster privilege.

User API

AddingSubmit a PUT or POST request to the /_shield/user/ endpoint

Response

{ "user": { "created" : true }}

POST /_shield/user/alice{ "password" : "j@rV1s", "roles" : [ "user", "other_role1" ]}

If you are updating a user, you can omit the password field unless you want to change the users password. You must specify the users roles. Omitting the optional full_name, email, or metadata fields sets those values to null.

User API

RetrievingGET request to the /_shield/user endpoint

Deleting DELETE request to the /_shield/user/ endpoint

DELETE /_shield/user/ironman

GET /_shield/user

GET /_shield/user/alice,bob

Managing Roles

Roles ApiAdding RolesPUT or POST request to the /_shield/role/ endpoint

POST /_shield/role/my_role{ "cluster": ["monitor"], "indices": [ { "names" : [ "index1", "index2" ], "privileges" : ["all"], "fields" : [ "title", "body" ], "query" : "{\"match\": {\"title\": \"foo\"}}" } ] }

In the role definition, you specify the cluster privileges and indices privileges the role grants.

Optionally, you can also configure field and document level security and assign run_as privileges.

Shield Privileges

Cluster privileges

all

monitor

manage

manage_security

mangae_index_templates

transport_clients

Managing Roles

All cluster operations, like snapshotting, node shutdown/restart, settings update, rerouting, or managing securityAll cluster read-ony operations, like cluster health & state, hot threads, node info, node & cluster stats, snapshot/restore status, pending cluster tasksBuilds on monitor and adds cluster operations that change values in the cluster. This includes snapshotting, updating settings, and rerouting. This privilege does not include the ability to manage security.All security related operations such as CRUD operations on users and roles and cache clearingAll operations on index templatesAll privileges necessary for a transport client to connect

2. Indices privileges all

monitor

manage

read

index

create

delete

write

delete_index

create_index

Managing Roles

Any action on an indexAll monitor privileges plus index administration (aliases, analyze, cache clear, close, delete, exists, flush, mapping, open, force merge, refresh, settings, search shards, templates, validate, warmers)All actions, that are required for monitoring and read-only (recovery, segments info, index stats & status)

Read only access to actions (count, explain, get, mget, get indexed scripts, more like this, multi percolate/search/termvector), percolate, scroll, clear_scroll, search, suggest, tv)

Managing Roles

Response

Can also use wildcards and regular expression to reffer to multiple indices.

{ "role" : { "created" : true }}

"foo-bar": # match the literal `foo-bar`"foo-*": # match anything beginning with "foo-""/.*-201[0-9]-.*/": # match anything containing 2010-2019

Regular Expressions - to match more complex patterns you can enable regular expressions by wrapping the pattern within a pair of forward slashes (/).

Regular expressions are based on Lucenes regexp automaton syntax. Any pattern starting with / and not ending with / is considered to be malformed.

Managing Roles

Example:

POST /_shield/role/customer_care { "indices": [ { "names": [ "*" ], "privileges": ["read"], "fields": [ "issue_id", "description", customer.*

] } ]}

Important

Document and field level security is currently meant to operate with read-only privileged accounts. Users with document and field level security enabled for an index should not perform write operations.

Managing Roles

Retrieving RolesGET request to the /_shield/role endpoint:

Deleting RolesDELETE request to the /_shield/role/ endpoint

GET /_shield/role

GET /_shield/role/my_admin_role,log_admin_role

DELETE /_shield/role/my_admin_role

If the role is successfully deleted, the request returns {"found": true}. Otherwise, found is set to false.

Node Authentication and Channel Encryption

With Shield, you can use SSL/TLS to encrypt communication to and from nodes.

When SSL/TLS is enabled, the nodes validate each others certificates, establishing trust between the nodes.

Require that nodes authenticate new nodes that join the cluster using SSL certificates.

Enable Message Authenticaton

Message authentication verifies that a message has not been tampered with or corrupted in transit during node-to-node communication.

This creates a system key file in CONFIG_DIR/shield/system_key.Copy the genererated system key to the rest of the nodes in the cluster.

Notes: The system key is a symmetric key, so the same key must be on every node in the cluster.

Bin/shield/syskeygen

Enable Auditing

It allow you to stores a record of attempted and successful interactions with your Elasticsearch cluster. You can use this information track of who is doing what to your cluster and identify potential security issues.

To enable auditing, add the following setting to elasticsearch.yml:

By default, events are logged to a dedicated elasticsearch-access.log file in ES_HOME/logs.

shield.audit.enabled: true

You can also store the events in an Elasticsearch index for easier analysis and control what events are logged. For more information, see Configuring Auditing.

shield.audit.outputs: [index, logfile]

Note

Audit events are batched for indexing so there is a lag before events appear in the index. You can control how frequently batches of events are pushed to the index by setting shield.audit.index.flush_interval in elasticsearch.yml.

Introduction

Kibana is an open source analytics and visualization platform designed to work with Elasticsearch

Use Kibana to search, view, and interact with data stored in Elasticsearch indices.

Easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.

Easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

Features

Seamless Integration with ElasticsearchArchitected to work with Elasticsearch, Kibana gives shape to any kind of data structured and unstructured indexed into Elasticsearch. It also benefits from Elasticsearch's powerful search and analytics capabilities.

Give shape to your dataTo better understand large volumes of data, easily create bar charts, line and scatter plots, histograms, pie charts, and maps.

Sophisticated AnalyticsLeverage the power of Elasticsearch analytics capabilities to analyze your data intelligently, perform mathematical transformations, and slice and dice your data as you see fit.

Flexible Interface, Easy to ShareEasily create, save, share, and embed your visualized data for quick and smart communication.

Features

Easy SetupSimple and friendly setup and startup. Kibana 4 ships with its own web server to help you get up and running quickly.

Visualize Data from Many SourcesEasily visualize data pushed into Elasticsearch from Logstash, ES-Hadoop, Beats, or third-party technologies like Apache Flume, Fluentd, and many others.

Simple Data ExportEasily export interesting bits of data to merge and meld with other data sets to quickly prototype new analyses and discover something new.

Installations

To get Kibana up and running:

Download the Kibana 4 binary package for your platform.

Extract the .zip or tar.gz archive file.

Run Kibana from the install directory: bin/kibana (Linux/MacOSX) or bin\kibana.bat (Windows).


Kibana users have to authenticate when your cluster has Shield installed.

Need to configure credentials for the Kibana server to authenticate requests coming from Kibana webserver.

To use Kibana with Shield:

Configure credentials for the Kibana server.

Create a user account for the Kibana server and assign it the kibana4_server role

esusers useradd kibana4-server -r kibana4_server -p password

The Kibana server needs access to the cluster monitoring APIs and the .kibana index. The server does not need access to user indexes. The required privileges are specified in the kibana4_server role provided in the default Shield roles.yml


2. Specify the credentials for your Kibana server user in the Kibana configuration file, /config/kibana.yml.

elasticsearch.username: "kibana4-server"elasticsearch.password: "password"

Kibana users need access to the indices that they will be working with and the .kibana index where their saved searches, visualizations, and dashboards are stored.

Users also need access to the kibana-devnull index


3. Derive Kibana user roles

kibana_user: cluster: - monitor indices: - names: 'customer' privileges: - view_index_metadata - read - names: '.kibana' privileges: - manage - read

the following kibana_user role only allows users to discover and visualize data in the customer indices.


4. Assign the appropriate roles to your Kibana users or groups of users

POST /_shield/user/alice { "password" : "t0pS3cr3", "roles" : [ "kibana_user" ]

}

Kibana users need access to the indices that they will be working with and the .kibana index where their saved searches, visualizations, and dashboards are stored.

Users also need access to the kibana-devnull index


5. Install the Shield plugin into Kibana. The Shield plugin secures user sessions and enables users to log in and out of Kibana.

6. Configure Kibana to encrypt communications between the browser and the Kibana server.

bin/kibana plugin --install kibana/shield/2.3.3

server.ssl.key: /path/to/your/server.keyserver.ssl.cert: /path/to/your/server.crt

shield.encryptionKey: "something_secret"

Once you enable SSL encryption between the browser and the Kibana server, access Kibana via HTTPS. For example, https://localhost:5601.

Set the shield.encryptionKey property in the kibana.yml configuration file. You can use any text string as the encryption key.

Kibana Dynamic Mapping

Kibana needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .kibana index where saved searches, visualizations, and dashboards are stored.

Create the .kibana index with dynamic mapping enabled just for that index:

PUT .kibana { "index.mapper.dynamic": true }

References

https://www.elastic.co/products/kibana

https://www.elastic.co/products/shield

http://blog.trifork.com/2015/03/05/shield-your-kibana-dashboards/

https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-2-discover/

https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-3-visualize/

https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-4-dashboard/

Thank You