from threat intelligence to rap sheets (in just a … 1/dragos...•finding other domains sharing...

1© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

FROM THREAT

INTELLIGENCE TO RAP

SHEETS

(IN JUST A FEW STEPS)Dragos Comaneci, Principal Software EngineerAdrian Hada, Security Research Engineer

2© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

ABOUT

• Software Engineer, Ixia

• Loves to build, debug & understand

distributed systems

• Security Research Engineer, Ixia

• Spends most of his time around malware,

botnets and the like

3© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

AGENDA

• What’s a Rap Sheet?

• Threat Identification

• Storing and Interpreting Data

• Stats

4© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

DISCLAIMER

• Not product placement

• Our perspective on developing a threat intelligence system

5© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

WHAT’S A RAP SHEET?

• “a list kept by the police of all the times a

person has been arrested” (m-w.com)

• an official police document that lists the

crimes that a particular person has

committed (dictionary.cambridge.org)

6© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

WHAT’S A RAP SHEET?

• Expanding upon the

idea

• Track all malicious

IPs on the Internet

over time

7© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

WHAT’S A RAP SHEET?

• IP address or domain

• Proof of maliciousness

• 100% certainty

• No moral judgement

8© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

WHAT’S A RAP SHEET?

• Malware is dirty

• Exploit kits are dirty

• Bots exploiting vulnerabilities are dirty

• Phishing pages are dirty

• Spam is “clean”

• pr0n is “clean”

Examples

9© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 9© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION (FINDING

THE BADDIES)

10© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Battery of AV products

• Threat intelligence feeds

• Detection threshold

• Lower chance of False Positives

Virus Scanning

11© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• No execution

• Interesting properties/artefacts

• Imported/mentioned functions

• Sections

• Entropy

• Certificates

• Particular strings

• Children’s game using Mimikatz

• Educational app dumping passwords

• Other intel on binary

Static Analysis

12© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Cuckoo Sandbox

• Great project!

• VM/sandbox hardening is a must

• Cuckoo does some of this work for you

• VMs are easy to revert and reuse

• (alternatively) Execute on hardware – slow cleanup

• Analyze behavior on execution

• All sorts of honey

• Applications, documents, credentials

• Grab more Intel – dropped files, contacted IPs, URLs, etc

Dynamic Analysis

https://xkcd.com/694/

13© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Responsibility!

• Try not to:

• Spam others

• Gathering spam is also useful

• DoS others

• Brute force others

• Infect others

• Sometimes more easier said than done!

Dynamic Analysis

14© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Multiple honeypots

• Dionaea, Kippo/Cowrie, Glastopf and others

• Globetrotting

• Different continents

• Different countries

• Different provider sizes

• Hard to administer!

Honeypots

Internet

15© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Learning from the enemy

• Honeypot proxies

• Forward to real honeypot

• Easy to deploy/redeploy

• No dependency/OS issues

• Honeypot blacklisting

• One jar for many flies

• Dump to central repository

Honeypots

Internet

Honey courtesy of Jeff Geerling and http://cliparts.co/

16© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• False positives

• Scanners – malicious or not?

• Indexing bots – real or fake?

• Only identify attacks

• Signatures for attacks

Honeypots

17© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATIONPhishing Detection

18© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Static “signatures” for larger targets

• Plenty of challenges

• Signature development

• No false positives

• Originals will always match

• Phish of phish of phish…

• Redo periodically

• Limited detection

• Easy to bypass

• Must look Facebook-y or Google-y, not exact clone

Phishing Detection

19© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Generic, machine learning-based approach

• Processes the HTML code

• Tries to classify correctly

• False positives likely

Phishing Detection

20© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

THREAT IDENTIFICATION

• Hostnames, domains and IP addresses

• Valuable information

• Important infrastructure services (whitelisting)

• Reoccurrences

• Mapping threat actors

Passive DNS

21© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 21© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

HOW DO WE MAKE SENSE OF ALL

THE DATA WE’RE COLLECTING?

22© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

Rap Sheet System

STORING AND INTERPRETING DATAOverview

URLs

Binaries

Honeypot Attacks

Rap Sheets

Tracking Info:

• SHAs

• FQDNs

• IPs

• URLs

23© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATA

• Real-time system

• As soon as we have enough information to build a Rap Sheet, we build & publish it

• Dataflow model

• Each node receives some input and produces some output. E.g..

• Nodes are connected to one another in a Topology

• Nodes may interact with other external services (Databases, Storage, Sandbox execution, etc.)

• Special nodes

• Only produce output (e.g. scanning threat intelligence feeds and extracting URLs for processing)

• Only receive input (e.g. storing the final information into the database)

Some facts

URLs Fetcher Fetched contents

24© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATA

• Each node does one small thing (microservices anyone ? )

• Easy to develop & test

• Easy to reuse

• Easy to reason about

• Nodes can be combined in different ways in a topology

• Nodes can be scaled individually

• New nodes can be easily integrated

• Each node can be updated individually

Why this model ?

25© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATA

• Blob data

• Fetched URL contents, packet captures, dropped files, sandbox analysis results, screenshots, etc.

• Cloud storage (currently AWS S3)

• Structured data

• NoSQL multi-model DB called ArangoDB

• Key/value

• Document

• Graph

• Schemaless

• JSON objects

• Easy to add new fields & information

• AQL, transactions, indexes, joins

Where do we store data ?

26© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATAHow do we store things?

FQDN IP

URLSHA

Resolves to

Fetches to

• Four main entities with

associated information

• Graph links between them with

more information

27© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATA

• Graph naturally maps to the underlying problem domain

• We can do interesting queries like:

• Finding all IPs that have served a certain malicious SHA

• Finding redirector domains (& URL shortening services)

• Finding other domains sharing the same NS/MX servers

(as well the the usual PassiveDNS type queries)

• Finding IPs & Domains which served SHAs which were dropped during dynamic analysis by SHAs

coming from a specific IP.

• Flexibility:

• Can easily tack on new information & entities to the Graph

Why do we store it this way?

28© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STORING AND INTERPRETING DATA

• Tech

• Datacenter OS & Apache Mesos

• Simple deployment for lots of distributed services (Redis, ELK, RabbitMQ, ArangoDB, Storm, etc.)

• Containerize all the things and then run them using Marathon

• Apache Storm

• Principles

• Split up your components

• Split up your database

• Common and battle hardened infrastructure components

• Moving code to data instead of the other way around

Scaling it up

29© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 29© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STATS

30© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STATSActive Rap Sheet Age

011

17

23

29

35

41

47

53

59

65

71

77

83

89

95

10

11

07

11

311

91

25

13

11

37

14

31

49

15

51

61

16

71

73

17

91

85

19

11

97

20

32

09

21

52

21

22

72

33

23

92

45

25

12

57

26

32

70

Number of Rap Sheets still considered malicious by age IP Count by Age

31© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STATSDelisted Rap Sheet Age

01

32

12

73

33

94

55

15

76

36

97

58

18

79

39

91

05

111

11

71

23

12

91

35

14

11

47

15

31

59

16

51

71

17

71

83

18

91

95

20

12

07

21

32

19

22

52

31

23

72

43

24

92

55

26

12

71

Time an IP Continues to Behave Maliciously IP Count by Age

32© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |

STATS

• Top 20 – 85%

• “Usual suspects” – US, China, Germany

• Surprising – Vietnam, Indonesia

Malicious IP Addresses by Country

US35%

CN10%DE

5%

FR5%

RU4%

VN3%

BR3%

IT2%

GB2%

PL2%

UA2%

NL2%

KR1%

HK1%

IN1%

TR1%

MX1%

ID1%

AU1%

OTHER17%

IP Count By Country

US CN DE FR RU VN BR IT GB PL

UA NL KR HK IN TR MX ID AU OTHER

33© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |