● twenty years experience in security including experience as a cloud security lead in one of the ten largest AWS environments

● patent holder; published researcher; advisor to various security product plays and VCs; credited bug hunter

● veteran of six startups including two successful exits● contributed, as an architect and / or core business logic

developer, to three successful security products, and six large-scale security monitoring and threat hunting projects, in both cloud and legacy environments

Craig Chamberlain | @randomuserid

Intro➢This is a talk about doing intrusion detection in the cloud

○And what I have learned➢This is a talk about technologies, not products

○Specification based vs. behavioral intrusion detection

➢This is “pure” research○No product vendors had editorial control

Lessons Learned: 2016 - 2018

Lessons Learned

➢ Network intrusion detection is relatively hard & expensive to do○ In the cloud

➢ When you do get it working, the results are not what you expected○ Results were better from behavioral detection using

endpoint data - then from network data➢ FOSS (free / open source software) alternatives deserve

serious consideration

Lessons Learned

➢ Network intrusion detection is relatively hard & expensive to do○ In the cloud

➢ When you do get it working, the results are not what you expected○ Results were better from behavioral detection using

endpoint data - then from network data➢ FOSS (free / open source software) alternatives deserve

serious consideration

Network IDS Assumptions1. The monitored network is not already compromised2. We can detect attacks by collecting network traffic and

applying packet signature matching rules3. Attacks are rare and finite enough to be quantifiable4. We can enumerate all possible attack methods and

create a dictionary of signatures to describe attack patterns

5. We can maintain signature convergence in advance of emerging threats

6. We can negate all conceivable evasion methods

What Is Intrusion Detection?➢ Wikipedia: “An intrusion detection system (IDS) is a device or software

application that monitors network or system activities for malicious activities or policy violations and produces electronic reports to a management station.”

➢ SANS: ‘Intrusion Detection can be defined as "...the act of detecting actions that attempt to compromise the confidentiality, integrity or availability of a resource.” More specifically, the goal of intrusion detection is to identify entities attempting to subvert in-place security controls.’

What Is Intrusion Detection?➢Network vs. Host Intrusion Detection

○Network intrusion detection can be performed by a device (in a network) or a network filter driver (on a host.)

➢Open vs. Closed Source○Snort (open source IDS) and its derivatives (Suricata)○Commercial closed-source products

Network Layer Intrusion Detection➢ In the “real” world, this is done by specialized software

and hardware (think ASICs)○The PC architecture was largely abandoned long ago○In order to process 1/10 GB streams with low latency

➢There are no hardware firewall / IDS devices in the Cloud (that we can touch)

Introspection? Nope.

Cloud NIDS Design Patterns➢vNetwork based

○Inline vNIDS device cluster○Passive vNIDS cluster○Backhaul to on-prem hardware

➢ Instance / guest based○IDS agent with a network filter driver

InlineAppliancevNIDSDesignPattern

Direct Connect / VPN On-Premise Datacenter

Firewall / IDS Cluster

Internet

PassivevNIDSDesignPattern

What About Containers?

➢ None of these patterns can inspect east-west traffic between containers

➢ East-west traffic may be the one place to see unencrypted network activity○ Most user-facing services run over TLS in the

post-Snowden / encrypt everything era➢ This creates enormous blind spots

Blind Spots

Guest / Instance Based IDS➢ Advantages

○Simplicity (assuming one agent)○Superior vantage point

➢Disadvantages○Performance tuning constraints gate signature, rule use; sometimes leads to agent sprawl

○Performance impact may be unacceptable

InstanceBasedvNIDSDesignPattern

Lessons Learned

➢ Network intrusion detection is relatively hard & expensive to do○ In the cloud

➢ When you do get it working, the results are not what you expected○ Results were better from behavioral detection

using endpoint data - then from network data➢ FOSS (free / open source software) alternatives deserve

serious consideration

top - 18:01:13 up 6:20, 3 users, load average: 53.42, 53.06, 45.76Tasks: 424 total, 53 running, 371 sleeping, 0 stopped, 0 zombieCpu(s): 80.8%us, 14.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.6%si, 0.0%stMem: 7399012k total, 3800740k used, 3598272k free, 213076k buffersSwap: 0k total, 0k used, 0k free, 2086332k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5230 root 20 0 1046m 565m 7728 S 66 7.8 20:28.55 Suricata-Main 29038 piglet 20 0 422m 16m 8464 R 3 0.2 0:04.44 php-fpm 29045 piglet 20 0 424m 17m 8464 R 3 0.2 0:04.48 php-fpm 29390 piglet 20 0 423m 16m 8456 R 3 0.2 0:04.50 php-fpm 29480 piglet 20 0 424m 17m 8456 R 3 0.2 0:04.41 php-fpm 29744 piglet 20 0 424m 17m 8456 R 3 0.2 0:04.36 php-fpm 29761 piglet 20 0 423m 16m 8456 R 3 0.2 0:04.31 php-fpm

25

Performance Impact

Testing Methodology1. Start with the top of the kill chain (reconnaissance) and

work down to the more specific2. Select particularly relevant exploit(s) from local history

a. Non-famous exploits without logos or marketingb. With critical impact (e.g. RCE)

3. Continue down the kill chain to consider all relevant threat classes, not just IDS test cases

Interactive Section: Threat Hunting: Network vs. Endpoint Data

Example One: One of these network connections is a persistence mechanism.

Which one?

What is netcat?The “swiss army knife” for networks

Source: https://twitter.com/b0rk/status/1059651145461723139

Example Two: One of these connections is data exfil (a

data leak.)Which one?

➢ IOCs: The Fundamentals○Anomalous user activity

■Privilege elevation■Suspicious session / authentication events■Anomalous command activity

○File Integrity Monitoring■Anomalous system file writes

○Anomalous process activity

2. We can detect suspicious activity by collecting network traffic and applying packet signature matching rules?

6. We can negate all conceivable evasion methods?

➢

○○○○

..threat actors tend to do the unexpected, launch attacks we had not thought of.

Lessons Learned

➢ Network intrusion detection is relatively hard & expensive to do○ In the cloud

➢ When you do get it working, the results are not what you expected○ Results were better from behavioral detection using

endpoint data - then from network data➢ FOSS (free / open source software) alternatives

deserve serious consideration

The Future is Open➢Open Source SIEM (security information and event

manager)○Using ELK (Elasticsearch, Logstash, Kibana)

■Log aggregation, security analytics and threat hunting

○OSS intrusion detection (Suricata)○Behavioral intrusion detection

■Semi-agentless instrumentation■Auditd (Linux), Sysmon (Windows)■Web service, system logs

Open Source Alternatives

Lessons Learned

➢ Network intrusion detection is relatively hard & expensive to do○ In the cloud

➢ When you do get it working, the results are not what you expected○ Results were better from behavioral detection using

endpoint data - then from network data➢ FOSS (free / open source software) alternatives deserve

serious consideration

EpilogueThe network vs. endpoint data debate: advocating abandoning network data in favor of endpoint data.

Don’t do this!

The Data TrinityAPI logs (cloudtrail, stackdriver, etc)

Network data (flows, IDS alerts, packets, firewall / proxy events)

Workload data - servers, containers, apps, (serverless) functions - wherever compute happens

Machine Learning: MetronMetron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.

http://metron.apache.org/

What would you like to see?

Craig Chamberlaincraig@spacecake.io

@randomuserid