hids as a service
TRANSCRIPT
HIDS as a Service
Ivan AgarkovSecurity Infrastructure Engineer
20k HIDS cluster definitive guide
About myself
- Ivan Agarkov- 2003-present - security guy- Securing Wargaming since 2015- SELinux & Perl fan- Internal trainer- Doing ‘security RnD’- ‘Extreme’ CTF tasks author
@annmuor
What the f***ing HIDS?
• H - Host• I - Intrusion• D - Detection• S - System
What does HIDS mean?
collect
How it works?
analyze
detectalert
Collection
logs
file checks
active checks
rootkit checks
normalize data compress data send for analysis
Detection
log classify
generate eventfill meta
set priority
ruleset check
Analysis
● How many times an event fired?
● What was changed since the last run?
● Is it eligible to generate an alert?
● Is it eligible to set alert as ‘multiple’?
Alert
alert store / archive
send report( later ) notify
urgent?email
messenger
phone call
escalation
OSSEC-related
collect ossec-agentlessd
ossec-logcollector
ossec-syscheckd
ossec-agentd
ossec-remoted
detect &analyze
ossec-analysisd
alert ossec-reportd
ossec-maild
ossec-integrated
A long time ago, in a galaxy far far away...
2010 - 2014
- 50 - 5000 servers- Manual log handling- syslog + ansible to
collect- cat/grep to find
something- how did we live?
like that!
2014 - 2016
- 5000-10000 servers- ELK stack to collect
logs- Kibana to find
something- What could go wrong?
136M logs ( strings ) per day, oops
2016 - present
- 10k-20k servers- HIDS agent on each server- Collect only significant- Alert if something goes
wrong- Kibana is still here
200-300k events per day ( now )
Building the cluster
First try
ossec central
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
First try - results
osseccentral
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
databaselosing data
UDP
Second try
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
Second try - results
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
database
UDP
bottle neck
Switched to WAZUH
=
Third try
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
UDP
elasticlogstash
Third try - results
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
UDP
udp overload
elasticlogstash
Third try - details
UDP TCP
Finally
ossec@dc ossec@dc ossec@dc
nodesnodes nodes
TCP
elasticlogstash
Data collection
Data collection scheme
ossec cluster
elasticlogstash wg plugin
redis
ossec rulesetlogs
kibana
OSSEC ruleset● Based on wazuh PCI DSS ruleset● Works as puppet submodule● Alerts count was reduced 20 times● 60% of ruleset is useless● Custom rules based on our needs● Reduces logs 450 times !Collect 1
week stats
Lower levels
Find useless
6 000 000 3 000 000 300 000 in 3 month
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
WG plugin
● Put server’s metadata into the alert● Put user’s metadata into the alert● Normalize alert’s data● Hides secret data
server responsible
user real name
ssh key owner
server owner
alert
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
WG plugin/redisossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
ssh keys archive
CMDB
redisSLOW FAST
WG plugin/sample
vs
Elasticsearch
nfs data node
node1 node2
curator jobs
kibana
logstash output
● Alias per project/owner● Archive old indexes
SOC network
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
Elasticsearch/curator
● Runs once per day● Creates ‘aliases’● Hides some data from teams● Prevents information disclosures
Last but one boring scheme
thing 2
thing 1
database
Kibana
kibana
● ADFS + mod_mellon to authenticate● nginx + mod_lua to authorize● user groups = server groups = aliases
User network
apachemod_mellon
nginxmod_lua
ADFS user groups
ossec cluster
elasticlogsta
shwg
plugin
redis
ossec ruleset
raw logs
kibana
Command & Control
Server lifecyclesetup
ready
production shred
free
HIDS lifecycle
production shred
Production
● Install HIDS agent● Find HIDS server● Do a handshake● Download agent configuration● Start agent service● Deal with failures
HIDS puppet code package
$server from hiera
config.erbagent-auth
service
zabbix logrotate
Deal with failures
● Service failed puppet failed● No logs from agent zabbix trigger● Port is down zabbix trigger
Shred
● Remove host from HIDS server
Remove host from HIDS server
● Each server is running Wazuh API● API allows to manage agents● Cleanup agents on shred● Cleanup agents on ‘connection loss’
Making profit
How can we help engineers?
Track users
Debug SELinux
Figure out how new feature breaks our web
Find puppet bugs
Look for hacking attempts
How can we help business?
Control our employees
Generate reports & trends
Inform about significant events
Create annual reports
How to get more?
Take my money!
More money!
No more money :(
Afterword
- Worth it?- Sure- Will help to secure my business?- Indirect ways mostly
