wrangle 2016: malware tracking at scale

Malware Tracking at Scale

About me• Michael Bentley• Formerly Director of Research and Response @ Lookout• Currently working on data mining projects• KK6WCN• michael@setnorth.com

Agenda• What we are trying to accomplish• How basic heuristics work• Where basic heuristics don’t work• Tracking with pairwise similarity and EMR• Visualizations to help extract more information• Mistakes and caveats

What are we trying to accomplish• Searching for major versions of software (malware)• Find ways to detect it with simple heuristics• Find ways to track it• Dataset discovery

Simple heuristics• Detect on static data• Detect on analysis stack created metadata

applications

analysisacquisition

Hashes

Strings

Who signed it / certificate

Simple heuristics - hashes

APK file

Hashes Icon

Dex File

Simple heuristics - string detection

• Nice ASCII string delimited by null bytes

• Malicious class path• Byte code• Exact match in one or both

directions of string• Ctrl + F

Null byte

Simple heuristics- certificates• Same

malware• Different

certificates

Where simple heuristics are good• Good for things that don’t change• Computationally cheap• About the same scenario for network (IDS) or

application inspection (malware detection)

Where it’s problematic• Anything with funding/making money.• Malware created in Eastern Europe, Asia, Italy

(Hacking Team)• Mass creation of certificates• Code taken from Stack Overflow

• Anything with basic string obfuscation• Hunting for new major versions

Enter pairwise similarityYou’re about to see a spreadsheet at a big data conference

http://gunshowcomic.com/648

Application pairwise similarity

Go from pick one app and rescan corpus

Pick one application – Rescan corpus• Examine one app• Find heuristic• Rescan corpus• Rinse repeat ad infinitum• Throw people at the problem

http://bit.ly/2a0zcZR

Decoding what you already have• Pairwise similarity defines the

relationships for us• Dots represent unique (SHA1)

applications• Colors represent major versions

of malware• Each color is within ~85% match of

code distance

Clustering and intelligence

APKNearest neighbor

95% similar

Cluster 185% similar

Cluster 285% similar

Cluster 0< 85% similar

• APKs are nodes and edges• Clusters are neighborhoods

Clustering and intelligence

Clustering versus heuristics

Evolution of malware over time• By taking the clustering data

and then overlaying it with the packaged at data we can watch malware evolve over time.

• Color represents major version• Time is a 4 month sliding

window• Shows iterations from malware

writers

Pairwise problems and options• Comparing 3500 applications is 12,250,000 operations• As you bring more applications in, expect to scale EMR

cluster or reduce n.• You can overmatch on similarity – outlier issue

Tripping over the bar• Pairwise similarity for 7k apps is about

5gB.• So is S3• Things go bad when you don’t respect the

bucket size• Troubleshooting CSV sizes is a thing

• Doesn’t work well on small applications• Temporary files on your local machine

that are 70gB cause problems

Knowledge• I had never used NetworkX before ~2014• I had no idea how to go from what we had into a decent format for

visualizing this (GraphML).• Almost no experience in graph theory before ~2014• Gilad Lotan had a great PyCon talk which got me started. I still reference

his talks.• Gephi is a great shortcut for visualizing in 2D if you aren’t familiar with

D3• Seth Hardy who gave tons of amazing feedback while I was learning• Jack Urban who proved that it was possible to track applications as a

network• Gensim library is a great way to get started in doing comparisons of

applications• Lots of inspiration from the Defcon 22 OpenDNS talk (theirs is better)

Thank you.

wrangle 2016: malware tracking at scale

Technology

malware slums: measurement and analysis of malware on...

pulse report: new apt32 malware campaign targets cambodian...

malware analysis analysis.pdf · definisi malware analysis...

wrangle the chute 2019 2 star results - march 7, 2019

subsurface geology of the area between wrangle hill and...

them's the rules - using a rules engine to wrangle...

helly hansen named title sponsor for wrangle the...

bsidescharleston2014 - ballin on a budget: tracking chinese...

d4.1 findings on economic modelling of malware as...

use of fancy bear android malware in tracking of ukrainian...

malware nilgün kablan. malware...

modern attacks and malware · cisco advanced malware...

improving accuracy of malware detection by …...ffri, inc....

wrangle the tech!

shmoocon 2015: no budget threat intelligence - tracking...

building attractors that wrangle leads

wrangle 2016: driving healthcare operations with small data

london times tables rock wrangle - amazon s3 times tables...

wrangle 2016: growth of iot, in numbers

malware analysis and antivirus technologies: kernel malware...