Download - DEF CON 23 - NSM 101 for ICS
![Page 1: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/1.jpg)
NSM 101 for ICS
![Page 2: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/2.jpg)
About me
Chris Sistrunk, PE Electrical Engineer Sr. ICS Security Consultant
– Control system security assessments – ICS Village (DEF CON & RSA Conference)
Entergy (11+ years) – SCADA Engineer (10 years) – Project Robus (ICS Protocol Fuzzing)
• 30+ implementation vulnerabilities in DNP3 stacks – Substation Security Team
BSidesJackson
![Page 3: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/3.jpg)
![Page 4: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/4.jpg)
What happens when you use nmap or a fuzzer on an ICS?
![Page 5: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/5.jpg)
If ICS are so vulnerable, why haven’t we seen
more attacks?
We aren’t looking!
![Page 6: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/6.jpg)
Two Key Reasons
Intent
Visibility
![Page 7: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/7.jpg)
Intent
Very little ICS targeted attack data Maroochy Shire to Stuxnet to German Steel Plant
Why are targeted attacks different? It’s a “Who” not a “What” Professional, organized, well-funded If you kick them out, they will return
![Page 8: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/8.jpg)
Visibility
![Page 9: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/9.jpg)
Visibility
![Page 10: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/10.jpg)
Public ICS Vulnerabilities Per Year
![Page 11: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/11.jpg)
If your ICS gets hacked…
gadgets water
electricity
you can’t make anymore
![Page 12: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/12.jpg)
Now what?
More Gov’t security regulations ICS security still lagging Breaches are inevitable Attacks aren’t stopping Every sector Including ICS
What can we do to get ahead of this???
![Page 13: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/13.jpg)
Network Security Monitoring
“The collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. NSM is a way to find intruders on your network and do something about them before they damage your enterprise.” - The Practice of Network Security Monitoring
![Page 14: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/14.jpg)
Network Security Monitoring
Invented in 1990, still in use today
Cliff Stoll “Stalking the Wily Hacker”
1988
Todd Herberlein et al.
“A Network Security
Monitor” 1990
US Air Force Defense
Information Systems Agency
Lawrence Livermore
National Lab Early 1990s
NetRanger RealSecure
Snort and many
others Late 1990s - early 2000s
Formal definition of
NSM 2002
![Page 15: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/15.jpg)
Before we start looking…
We need At least one person (to watch and hunt) The right tools to collect and analyze the data
![Page 16: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/16.jpg)
The NSM Cycle
Collection
Detection Analysis
Model for action, based on network-derived data
Requires people and process, not just technology
Focuses on the adversary, not the vulnerability
![Page 17: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/17.jpg)
Methods of Monitoring
Network tap – physical device which relays a copy of packets to an NSM sensor
SPAN or mirrored ports – switch configuration which sends copies of packets to a separate port where NSM sensor can connect
Host NIC – configured to watch all network traffic flowing on its segment (usually on NSM sensor)
Serial port tap – physical device which relays serial traffic to another port, usually requires additional software to interpret data
Fluke Networks Stratus Engineering
![Page 18: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/18.jpg)
Types of Data Collected Full content data – unfiltered collection of packets Extracted content – data streams, files, Web pages, etc. Session data – conversation between nodes Transaction data – requests and replies between nodes Statistical data – description of traffic, such as protocol
and volume Metadata – aspects of data, e.g. who owns this IP
address Alert/log data – triggers from IDS tools, tracking user
logins, etc.
![Page 19: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/19.jpg)
Difficulties for NSM
Encrypted networks Widespread NAT Devices moving between network segments Extreme traffic volume Privacy concerns
![Page 20: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/20.jpg)
Difficulties for NSM
Encrypted networks Widespread NAT Devices moving between network segments Extreme traffic volume Privacy concerns
Issues that most ICS do not face!
![Page 21: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/21.jpg)
Example ICS
Enterprise/IT
DMZ
Plant
Control
Web Historian or
other DB
DCS Historian HMI
PLCs, Controllers, RTUs, PACs
![Page 22: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/22.jpg)
Anatomy of an Attack
22
Over all Mandiant attack investigations, only a little more than half of victim computers have malware on them.
While attackers often use malware to gain an initial foothold, they quickly move to other tactics to execute their attacks.
Unauthorized Use of Valid Accounts
Known & Unknown Malware
Command & Control Activity
Suspicious Network Traffic
Files Accessed by Attackers
Valid Programs Used for Evil Purposes
Trace Evidence & Partial Files
![Page 23: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/23.jpg)
Attacker Objectives
Attacker’s goals: Damage equipment Affect or steal process info Cause safety or compliance issue Pivot from vulnerable ICS to
enterprise
Attacker’s options: Gain physical access to an ICS host Gain remote access to an ICS host Compromise a highly-privileged
client machine with access to the ICS network
Enterprise/IT
Plant DMZ
Control
Web Historian or
other DB
SCADA Historian HMI
PLCs, Controllers, RTUs, PACs
![Page 24: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/24.jpg)
Let’s do some NSM!
![Page 25: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/25.jpg)
Let’s do some NSM!
Inquisitive mind
NSM collection tools
NSM hunting tools
Protection
![Page 26: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/26.jpg)
NSM Collection
Firewall Logs Session Data NIDS/HIDS Logs Full packet capture Windows Logs and syslog SNMP (CPU % etc.) Alerts from security agents
(AV, whitelisting, etc.)
Enterprise/IT Enterprise collectors Logs and/or Agent
Network sensors Logs only
Plant DMZ
Control
Web Historian or
other DB
SCADA Historian HMI
PLCs, Controllers, RTUs, PACs
![Page 27: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/27.jpg)
NSM Collection
http://3.bp.blogspot.com/-B6PtheVJ9Jg/Uj4EErYhHdI/AAAAAAAAAFE/i_2dk9emrp4/s1600/Deer+tracks.jpg
![Page 28: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/28.jpg)
What are we looking for?
Exceptions from baseline (e.g. A talks to B but never C) “Top Talkers” Unexpected connectivity (to Internet, Business network) Known malicious IPs and domains Logins using default accounts Error messages that could correlate to vulnerabilities Unusual system and firewall log entries Host-based IDS or other security system alerts Unexpected file and firmware updates Antivirus alerts And others….
![Page 29: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/29.jpg)
NSM Detection & “Hunting”
Analyst looks at detected anomalies or alerts then escalates to IR
!
IDS alerts Anomaly detection Firmware updates, other
commands Login with default credentials High CPU or network bandwidth Door alarms when nobody is
supposed to be working Devices going off-line or behaving
strangely
Plant DMZ
Control
Web Historian or
other DB
SCADA Historian HMI
PLCs, Controllers, RTUs, PACs
![Page 30: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/30.jpg)
NSM Detection
http://www.buckmasters.com
http://www.jimyuskavitchphotography.com/data/photos/56_1wolf_track4.jpg
![Page 31: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/31.jpg)
NSM Analysis
Incident responders analyze the detected anomalies to find evil
Application exploitation Third-party connections (ex. ICCP
or vendor access) ICS-specific communication
protocol attacks (ex. Modbus, DNP3, Profinet, EtherNet/IP)
Remote access exploitation Direct network access due to poor
physical security USB-delivered malware
Plant DMZ
Control
Web Historian or
other DB
SCADA Historian HMI
PLCs, Controllers, RTUs, PACs
![Page 32: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/32.jpg)
NSM Analysis
http://alistairpott.com/wp-content/uploads/2008/05/rabbit-owl.jpg
http://www.youtube.com
![Page 33: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/33.jpg)
NSM Analysis
![Page 34: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/34.jpg)
NSM Analysis
![Page 35: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/35.jpg)
ICS NSM Examples
![Page 36: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/36.jpg)
Session Data “Top Talkers” FlowBAT characterizes Session Data, showing which nodes have the most traffic
Web traffic
Web traffic
NetBios
NTP
SiLK and FlowBAT can be easily installed in Security Onion
![Page 37: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/37.jpg)
Pcap Analysis for anomalies NetworkMiner can find potential ARP spoofing (as well as many other indicators)
![Page 38: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/38.jpg)
Pcaps - Abnormal DNS Traffic NetworkMiner sees“strange” DNS requests originating from within the ICS
![Page 39: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/39.jpg)
IDS alerts - Abnormal DNS Traffic DNS requests shown in the Bro IDS log in ELSA
![Page 40: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/40.jpg)
Pcaps – Malformed Modbus Deep packet inspection of Modbus by Wireshark
![Page 41: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/41.jpg)
Pcaps – Custom Modbus Unknown Function Code 90 Schneider Modicon uses FC 90 to start/stop the PLC and other admin stuff Metasploit module does too!
c
![Page 42: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/42.jpg)
IDS Logs
Bro IDS – DNP3 & Modbus – More ICS protocols being developed by UIUC
Snort IDS – DNP3 & Modbus preprocessors – ET SCADA & DigitalBond Quickdraw Snort rules
Suricata IDS – New DNP3 parser & ET SCADA rules
![Page 43: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/43.jpg)
IDS Logs
Modbus
DNP3
Bro IDS parses Modbus and DNP3 packets, ELSA consolidates Bro logs
![Page 44: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/44.jpg)
IDS GUIs Alerts in Sguil of scanning activity
![Page 45: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/45.jpg)
Syslog
Syslog can be configured to send to a NSM sensor or detected in network traffic if sent elsewhere. This is the Bro IDS Log for Syslog from an RTU.
![Page 46: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/46.jpg)
RTUs with Syslog
SEL-3530 RTAC GE D20MX Novatech OrionLX Cooper SMP 16
If not…require syslog and other logs in the ICS procurement language
![Page 47: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/47.jpg)
NSM Tools for the 7 Data Types Security Onion Linux distribution
– Easy to install and lots of documentation Full packet capture –
Tcpdump/Wireshark/NetworkMiner Extracted content – Xplico/NetworkMiner Session data – Bro/FlowBAT Transaction data – Bro Statistical data – Capinfos/Wireshark Metadata – ELSA (Whois) Alert data – Snort, Suricata, Sguil, Snorby
Peel Back the Layers of Your Network
![Page 48: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/48.jpg)
Security Onion Tools
![Page 49: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/49.jpg)
NetFlow Tools
SiLK & FlowBAT Install on Security Onion with 2 scripts www.flowbat.com
![Page 50: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/50.jpg)
Security Onion Implementation Test in a lab first Select suitable hardware platform More RAM is better Bigger hard drive is better (longer retention)
Mirrored/SPAN port on router/switch or a good network tap
Select proper placement of SO sensor The Practice of Network Security Monitoring Applied Network Security Monitoring
Work with the right stakeholders if placing in production
![Page 51: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/51.jpg)
Security Onion Implementation
I installed SO on an industrially hardened box SEL-3355 16GB RAM 1TB SSD Other boxes out there suitable for NSM sensor for industrial environments
![Page 52: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/52.jpg)
Security Onion Implementation
![Page 53: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/53.jpg)
SO for ICS = Security Ogre
![Page 54: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/54.jpg)
The Cuckoo’s Egg by Cliff Stoll https://www.youtube.com/watch?v=EcKxaq1FTac 1-hour NOVA Special (1990)
The Practice of Network Security Monitoring by Richard Bejtlich http://www.nostarch.com/nsm
Applied Network Security Monitoring by Chris Sanders & Jason Smith http://www.appliednsm.com/
The NSM Wiki http://nsmwiki.org http://securityonion.net
NSM References/Resources
![Page 55: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/55.jpg)
Takeaways
You can implement NSM in ICS today – without impacting your operations There are free tools available to
help you start looking at your ICS and hunting for evil
![Page 56: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/56.jpg)
People… …the most important part of NSM! Gigabytes of data and 1000s of
IDS alerts are useless without interpretation
Analyze data collected to understand what’s normal – and what’s not
Identify adversary TTPs and act to disrupt them
Remember Adversaries are a “Who”, not a “What”
![Page 57: DEF CON 23 - NSM 101 for ICS](https://reader034.vdocuments.net/reader034/viewer/2022052203/55d09973bb61eb765c8b4623/html5/thumbnails/57.jpg)
Greets
@ICS_Village – GO THERE! @dougburks & @securityonion team @chrissanders88 & @automayt @taosecurity @robertmlee for also banging the “NSM for ICS" drum @dan_scali & @robac3 My ICS peeps, SCADAbrothers, & SCADAsisters My love, @shannonsistrunk