mod_security + puppet + splunk

Prepared byPrepared by

WAF in scale v2.0

Alexey SintsovPrincipal Security Engineer26.02.2016

OWASP Delhi

22

Intro

© 2016 HERE | Security Monitoring System | SPC Engineering team

SPC Engineering

Team Product team 1

Product team 2

Product team 3

AWS US

Product team 3

Data Center 1 Data Center 2

Attack Surface (Internet/WEB)

AWS EU. . .

- A lot of different teams- Many different data centers (even AWS)- Only few security engineers- A lot of WEB attacks… SPC – Security Privacy Continuity

33

What we want:• We want to monitor WEB attacks, like IDS for WEB• We won’t review all script-kiddie/bot scans, we want auto confirmation and

correlation• We want to be able to do fast ‘virtual’ fixes in critical situations• We want SOC to be contacted when attack is confirmed (auto mode)

Þ We want WAF, but in monitor mode (until blocking is needed)

Additional needs:• We want to deploy and configure on all (if possible) FE - all products and

DataCenters• We want to control and update rules for those installations in “one way”• We want to make it “transparent” to avoid dependency on them• We do not want big performance impact on our services

Targets


44

Mod Security

• Detection Only mode• Only SPC rules for less CPU impact

Response based alerts:

1. Attack signature

2. Response Signature

• Parse response only if attack signature fired• If response signature fired -> True Positive alert!


55

Mod Security - simple example of rule:

SecRule REQUEST_BODY|REQUEST_URI|REQUEST_HEADERS "/+etc/+passwd“ "t:none,ctl:ResponseBodyAccess=On,msg:‘/etc/passwd request found…', phase:2,pass,nolog,auditlog,id:'950002',setvar:TX.ATTACK_ZLO=1, ctl:auditLogParts=+I,t:urlDecode,t:lowercase,severity:1“

…

SecRule RESPONSE_BODY "root\:x\:0\:0" "id:'950015',ctl:auditLogParts=+E, msg:'Content of /etc/passwd! (Rise incident to SOC)',phase:4, allow,nolog,auditlog,t:lowercase,severity:0"


66

Our Splunk app

• Correlation, analyses (we can code that!)• Search tool (incident analyses/analytics)• Alerting

Mail to 24/7 SOC

Call to oncall Security Engineer (Wake up!)


77

Design

Splunk forwarder

Apache - ModSecurity audit logs - SPC Rules

Attacks

Splunk serverHERE servers

SPC Splunk app

Search tool

SPC Engineer

index-security


88

Part of dashboard

© 2016 HERE | Title | Author | Company confidential

99

Attacker session analyses (correlation with Apache logs)


1010

Automated alerts


1111

Automated alerts (blind SQLi)


... | bucket _time span=30minutes | stats count distinct_count(length) as rl1 distinct_count(resplength) as rl2 values(response_stat) as "Response codes" by _time,hostname,ms_finalip, uri |eval delta=count/rl1 |where delta > 10 and count > 50 | ...

1212

THOR Integration exampleTHOR repo

HERE Rules

Apache

…

HERE server

Puppet agentyum install …

HERE Rules

Apache

Mod Security

Splunk forwarder

Configure all…

[1] http://www.netways.de/uploads/media/Pascal_Hahn_End_to_End_continuous_integration_of_deplayment-code_in_a_multi-tenant_puppet_setup.pdf

THOR:• Puppet as a service• Extensible & integration• Standard & building blocks

1


http://www.netways.de/uploads/media/Pascal_Hahn_End_to_End_continuous_integration_of_deplayment-code_in_a_multi-tenant_puppet_setup.pdf



1313

How it looks like for a product team


• Ask for a new server with: Apache, MySQL, PHP (not real case, just example)• Customer provides Puppet recipes of desired env. (Import MySQL schema, .htaccess rules,

PHP script deployment and etc) these two steps, THOR API and framework RPM• After deployment – our hardened server with configured Splunk, Apache and ModSecurity

and customer’s application, configs.• Customer’s tests (QA), including performance/stress

• If our ModSec provides not acceptable delay, than it will be found there• Ready to go! (with some minimal Security by default!)

- Customer does not have to think about WAF, configuration of logs and monitoring- If (s|)he adds new server, it will be automatically configured and will be under our monitoring

(in Splunk)

1414

Support

• Build server for ModSecurity rules (into RPM)• Automated unit tests for each rule

• Works as expected• Not blocking normal requests• No performance impact

• Version control via THOR API*

=> If new rule needs to be distributed, new RPM with new version will be tested and built (auto. mode).Then it can be updated via THOR API to new version for specific service or for whole env.

* We have more delivery platforms supported, but for this preso we are talking about THOR only


1515

If something goes wrong in OpenSource


1616

Summary• Maximum automation:

• Build new rules• Test new rules• Versioning• Auto Deploy• Auto alerting for REAL cases• Easy to investigate (evil POST requests)

• Good coverage:• All Apache (nginx) services – FE, WEB, RP, PVP• No dependencies on many different teams• Most common attacks and patterns – easy to do signatures, even for 0days

• Open Source , you can fix bugs by yourself!

- If we do not use Apache/Nginx?- If performance impact is too high for the service?- If it is not a WEB attack (HeartBleed)?- BASE64? Serialization? Specific attack vectors…


1717

THE FIN


WEB: https://www.here.com https://company.here.com/here/

twitter.com/asintsov [email protected]

https://www.here.com/

https://company.here.com/here/

