hids as a service

HIDS as a Service

Ivan AgarkovSecurity Infrastructure Engineer

20k HIDS cluster definitive guide

About myself

- Ivan Agarkov- 2003-present - security guy- Securing Wargaming since 2015- SELinux & Perl fan- Internal trainer- Doing ‘security RnD’- ‘Extreme’ CTF tasks author

@annmuor

i_agarkov@wargaming.net

What the f***ing HIDS?

• H - Host• I - Intrusion• D - Detection• S - System

What does HIDS mean?

collect

How it works?

analyze

detectalert

Collection

logs

file checks

active checks

rootkit checks

normalize data compress data send for analysis

Detection

log classify

generate eventfill meta

set priority

ruleset check

Analysis

● How many times an event fired?

● What was changed since the last run?

● Is it eligible to generate an alert?

● Is it eligible to set alert as ‘multiple’?

Alert

alert store / archive

send report( later ) notify

urgent?email

messenger

phone call

escalation

OSSEC-related

collect ossec-agentlessd

ossec-logcollector

ossec-syscheckd

ossec-agentd

ossec-remoted

detect &analyze

ossec-analysisd

alert ossec-reportd

ossec-maild

ossec-integrated

A long time ago, in a galaxy far far away...

2010 - 2014

- 50 - 5000 servers- Manual log handling- syslog + ansible to

collect- cat/grep to find

something- how did we live?

like that!

2014 - 2016

- 5000-10000 servers- ELK stack to collect

logs- Kibana to find

something- What could go wrong?

136M logs ( strings ) per day, oops

2016 - present

- 10k-20k servers- HIDS agent on each server- Collect only significant- Alert if something goes

wrong- Kibana is still here

200-300k events per day ( now )

Building the cluster

First try

ossec central

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

database

UDP

First try - results

osseccentral

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

databaselosing data

UDP

Second try

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

database

UDP

Second try - results

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

database

UDP

bottle neck

Switched to WAZUH

=

Third try

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

UDP

elasticlogstash

Third try - results

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

UDP

udp overload

elasticlogstash

Third try - details

UDP TCP

Finally

ossec@dc ossec@dc ossec@dc

nodesnodes nodes

TCP

elasticlogstash

Data collection

Data collection scheme

ossec cluster

elasticlogstash wg plugin

redis

ossec rulesetlogs

kibana

OSSEC ruleset● Based on wazuh PCI DSS ruleset● Works as puppet submodule● Alerts count was reduced 20 times● 60% of ruleset is useless● Custom rules based on our needs● Reduces logs 450 times !Collect 1

week stats

Lower levels

Find useless

6 000 000 3 000 000 300 000 in 3 month

ossec cluster

elasticlogsta

shwg

plugin

redis

ossec ruleset

raw logs

kibana

WG plugin

● Put server’s metadata into the alert● Put user’s metadata into the alert● Normalize alert’s data● Hides secret data

server responsible

user real name

ssh key owner

server owner

alert

ossec cluster

elasticlogsta

shwg

plugin

redis

ossec ruleset

raw logs

kibana

WG plugin/redisossec cluster

elasticlogsta

shwg

plugin

redis

ossec ruleset

raw logs

kibana

ssh keys archive

CMDB

redisSLOW FAST

WG plugin/sample

vs

Elasticsearch

nfs data node

node1 node2

curator jobs

kibana

logstash output

● Alias per project/owner● Archive old indexes

SOC network

ossec cluster

elasticlogsta

shwg

plugin

redis

ossec ruleset

raw logs

kibana

Elasticsearch/curator

● Runs once per day● Creates ‘aliases’● Hides some data from teams● Prevents information disclosures

Last but one boring scheme

thing 2

thing 1

database

Kibana

kibana

● ADFS + mod_mellon to authenticate● nginx + mod_lua to authorize● user groups = server groups = aliases

User network

apachemod_mellon

nginxmod_lua

ADFS user groups

ossec cluster

elasticlogsta

shwg

plugin

redis

ossec ruleset

raw logs

kibana

Command & Control

Server lifecyclesetup

ready

production shred

free

HIDS lifecycle

production shred

Production

● Install HIDS agent● Find HIDS server● Do a handshake● Download agent configuration● Start agent service● Deal with failures

HIDS puppet code package

$server from hiera

config.erbagent-auth

service

zabbix logrotate

Deal with failures

● Service failed puppet failed● No logs from agent zabbix trigger● Port is down zabbix trigger

Shred

● Remove host from HIDS server

Remove host from HIDS server

● Each server is running Wazuh API● API allows to manage agents● Cleanup agents on shred● Cleanup agents on ‘connection loss’

Making profit

How can we help engineers?

Track users

Debug SELinux

Figure out how new feature breaks our web

Find puppet bugs

Look for hacking attempts

How can we help business?

Control our employees

Generate reports & trends

Inform about significant events

Create annual reports

How to get more?

Take my money!

More money!

No more money :(

Afterword

- Worth it?- Sure- Will help to secure my business?- Indirect ways mostly

@annmuor

Questions?

i_agarkov@wargaming.net