Upload File

detecting malicious web links and identifying their attack ... · proposed a machine learning...

  • Home
  • Documents
  • Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various
25
2011-06-21 1 Hyunsang Choi, 2 Bin B. Zhu, 1 Heejo Lee 1 Korea University, 2 Microsoft Research Asia USENIX WebApps 2011 Detecting Malicious Web Links and Identifying Their Attack Types

Upload: others

Post on 17-Jul-2020

3 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

2011-06-21

1Hyunsang Choi, 2Bin B. Zhu, 1Heejo Lee1 Korea University,

2 Microsoft Research Asia

USENIX WebApps 2011

Detecting Malicious Web Links and Identifying

Their Attack Types

Page 2: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Outline

• Introduction• Existing solutions• Highlights of our approach• Discriminative features• Experimental results• Evadability• Conclusion

USENIX WebApps 2011 2011-06-21 Page 2

Page 3: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Webpages, Trustworthy?

blog.libero.it/matteof97

I want to read, … But is this Webpage safe to read?

Access or not access, that is a problem

2011-06-21 Page 3USENIX WebApps 2011

Page 4: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Malicious Webpages

Webpages have been widely used for malicious purposes

Growth of malicious URLs in 2010, Trend Micro Annual Threat Report, 2010

3 Major types of malicious URLs

2011-06-21 Page 4USENIX WebApps 2011

Page 5: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Existing Solutions: Blacklisting

The Achilles' heel of blacklisting

• Not work for new/unknown URLs• Evadable easily

Popular URL analysis tools

2011-06-21 Page 5USENIX WebApps 2011

Page 6: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Existing Solutions: Anomaly-Based Detection

• Other existing solutions: VM execution Rule-based detectors Machine learning based detectors

• Detecting typically a single type of an attack• Critical issues in machine learning based

approach What are highly effective discriminative features? Are the discriminative features en masse hard to

evade?

USENIX WebApps 2011 2011-06-21 Page 6

Page 7: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Highlights of Our Research Project

• Research Goals: Detect all major malicious types of URLs Identify attack types of a malicious URL Much harder than detection due to ambiguity

Develop effective & hard to evading discriminative features

• Methodology: machine learning based approach SVM for detecting malicious URLs RAkEL & ML-kNN for identifying attack types of a

malicious URL

USENIX WebApps 2011 2011-06-21 Page 7

Page 8: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Key Properties of Our Detector and Major Contributions

• First study to classify multiple types of malicious URLs

• A rich set of highly effective discriminative features Many features are novel and unique Same discriminative features for both detection and

classification tasks Robust against known evadsion techniques

• A systematical study of the effectiveness of each feature group

2011-06-21 Page 8USENIX WebApps 2011

Page 9: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Overview of Our System

6 groups of 53 discriminative features: Lexical Link popularity Webpage content DNS DNS fluxiness Network

31 out of the 53 features are novel or modified from prior arts

USENIX WebApps 2011 2011-06-21 Page 9

Page 10: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

1. Lexical Features

2011-06-21 Page 10

• Lexical features Most are targeted to detect phishing attack (phishing attack

has discriminate lexical property to deceive users) Discriminative features effective on some attack types but not

on other are desirable to distinguish different types

Targeted types

Phishing

Phishing

Phishing

Phishing

Phishing

Phishing

All types

Phishing

USENIX WebApps 2011

Page 11: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

2. Link Popularity Features

2011-06-21 Page 11

• Link popularity features Intuition: Malicious URLs are hardly indexed by normal users Methodology: Get inlink (incoming link) count from search

engines Search engines: AlltheWeb, Astalavista, Google, Yahoo, Ask

Targeted types

All types

All types

All types (SEO)

All types (SEO)

All types (SEO)

USENIX WebApps 2011

Page 12: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

2. Link Popularity Features (cont.)

2011-06-21 Page 12

• Blackhat SEO & link farming Blackhat Search Engine Optimization (SEO) is used to get

unethically higher search rankings Link farming: link manipulation using a group of webpages

to link together 5 features for detecting link manipulated URLs by Blackhat SEO

Distinct domain link ratio, max domain link ratio Spam, phishing, and malware link ratio

USENIX WebApps 2011

Page 13: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

3. Webpage Content Features

2011-06-21 Page 13

• Webpage content features Features used by Hou et al., “Malicious web content detection by

machine learning”, Expert Systems with Applications, 2010

Targeted types

Malware, phishing

Malware

Malware

All types

Malware, spam

Malware

Malware

USENIX WebApps 2011

Page 14: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

4. DNS Features

2011-06-21 Page 14

• DNS features Features from the DNS server Methodology: Use DNS answer data from DNS server

Targeted types

All types

All types

All types

All types

All types

USENIX WebApps 2011

Page 15: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

5. DNS Fluxiness Features

2011-06-21 Page 15

• DNS fluxiness features Features to detect fast-fluxing URLs Fast-flux: DNS technique to hide malicious websites behind an

ever-changing network of compromised hosts acting as proxies Methodology: Send queries to DNS server (first and consecutive

lookups) Features by Holz et al., “Detection and mitigation of fast-flux service

networks”, NDSS 2008

Targeted types

All types

All types

All types

All types

All types

USENIX WebApps 2011

Page 16: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

6. Network Features

2011-06-21 Page 16

• Network features Detect redirected URLs (URL shortening, iframe redirections) Methodology: Use web crawler

Targeted types

All types

All types

All types

All types

All types

USENIX WebApps 2011

Page 17: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Experimental Datasets

2011-06-21 Page 17

Single LabelURL Type

Single Label Dataset Amount

Benign

Randomly selected 20K URLs from DMOZ open directory 20K

Randomly selected URLs from Yahoo directory 20K

Spam jwSpamSpy list 11K

Phishing PhishTank list 4K

Malware DNS-BH list 17K

USENIX WebApps 2011

Page 18: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Evaluation Result – Detection Accuracy

2011-06-21 Page 18

• Detection accuracy 98.2% accuracy, 98.9% true positive rate, 1.1% false positive

rate, and 0.8% false negative rate

USENIX WebApps 2011

Page 19: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Evaluation Result – Link Popularity

2011-06-21 Page 19

• Link popularity Google reports a partial list of inlink information Without link popularity feature: 91.2% accuracy, 4.0% false

positive rate, and 4.8% false negative rate 90.03% accuracy in detecting link-manipulated malicious URLs

USENIX WebApps 2011

Page 20: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Datasets for Multi-Labels

2011-06-21 Page 20

• Datasets – Multi labels Use two website to crawl the ‘exact’ malicious type of URLs

(McAfee SiteAdvisor and Web Of Trust) About half of URLs in the data set have multiple labels

USENIX WebApps 2011

Page 21: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Evaluation Result – Multi-label Classification (1)

2011-06-21 Page 21

• Metrics Micro-averaged and macro-averaged metrics: Micro-average

gives equal weight to every data sets, while the macro-average gives equal weight to every category

Ranking-based metrics: Average precision and ranking loss• Multi-label classification result

93% averaged accuracy and 98% ranking-based precision

USENIX WebApps 2011

Page 22: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Evaluation Result – Multi-label Classification (2)

2011-06-21 Page 22

• Performance for each feature group No single feature group can effectively classify malicious URL

types

USENIX WebApps 2011

Page 23: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Evadability Analysis

2011-06-21 Page 23

• Robust to known evasion techniques Redirection: Network features Link manipulation: Link popularity features Fast-flux: DNS fluxiness features

• URL obfuscation IDN (Internationalized Domain Names) spoofing (e.g.,

www.pаypal.com = www.paypal.com)• JavaScript obfuscation

Deobfuscator• Social network sites

USENIX WebApps 2011

Page 24: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Conclusion

• Goal Proposed a machine learning approach to detect malicious

URLs and to identify attack types. • Method

Collect various types of discriminative features, detecting malicious URLs using SVM and identifying malicious URL types using RAkEL and ML-kNN

• Result Achieved an accuracy of over 98% in detecting malicious URLs

and an accuracy of over 93% in identifying attack types.• Contribution

Proposed several novel and highly discriminative features which provide a superior performance and a much larger coverage

First study to classify multiple types of malicious URLs, known as a multi-label classification

24 2011-06-21USENIX WebApps 2011

Page 25: Detecting Malicious Web Links and Identifying Their Attack ... · Proposed a machine learning approach to detect malicious URLs and to identify attack types. • Method Collect various

Q&A

25 2011-06-21USENIX WebApps 2011


Comprehensive Study of Selective Forwarding Attack in ... · of selective forwarding attack and malicious nodes. On ... Detecting Selective Forwarding Attacks in Wireless ... to detect

A Precise Algorithm for Detecting Malicious Sybil Nodes in ...ieiespc.org/include/pdfdownload.asp?filename=... · Abstract: A Sybil attack, where a malicious node creates multiple

Optimal Inspection Points for Malicious Attack Detection

Web security: an introduction to attack techniques and ... · Web security: basics Moving to web attacks Attack HTTP requests containing malicious payloads could attack web applications

Computer Security - Rutgers Universitypxk/419/notes/content/... · •Short URLs help mask malicious URLs –Redirect victims to a URL: looks like a legitimate Google accounts login

CYBER-ATTACK AGAINST UKRAINIAN POWER PLANTS...malicious software (malware) had been introduced into three Ukrainian power utilities via explicitly-targeted spear phishing attack (FBI,

Reaching Beyond The Perimeter...Email Security 3.0 Strategy | Deep Dive It is important to note that Mimecast can inspect and filter malicious URLs – via URL Protect for URLs embedded

e d i t i o n · Script-Based Attack Protection Detects attacks by malicious scripts that try to exploit Windows PowerShell. Also detects malicious JavaScripts that can attack via

URLDoc: Learning to Detect Malicious URLs using Online Logistic Regression Presented by : Mohammed Nazim Feroz 11/26/2013

Two Quarantine Models on the Attack of Malicious Objects

Optimal Inspection Points for Malicious Attack Detection ...tndinh/papers/cocoon14.pdf · Optimal Inspection Points for Malicious Attack Detection 3 { The center node rwhich represents

Invitation or Bait? Detecting Malicious URLs in Facebook ...precog.iiitd.edu.in/people/sonug/ic3.pdf · malicious Facebook posts in real-time. In another study, Dewan et al. identified

Analyzing and Blocking Malicious Traffic Based on Geolocation · 30 percent of malicious traffic, traffic actually qualified as a would-be suc-cessful attack, can be significantly

Introduction to Malicious Code (Malware) - Chalmers Maliciou… · Introduction to Malicious Code (Malware) ... • But what type of content is safe? ... Salami attack Usesseemingly

Detection and Prevention of Blackhole Attack in Wireless Sensor …ijsrcseit.com/paper/CSEIT1723234.pdf · 2017-06-06 · f) Blackhole Attack: In blackhole attack, a malicious node

Chapter 1 Viruses: Attack of the Malicious Programs

Cyber Liability and Malicious Attack

Lexical Mining of Malicious URLs for Classifying Android ...cse.unl.edu/~qyan/paper/Securecomm18_Lexical.pdf · Lexical Mining of Malicious URLs for Classifying Android malware Shanshan

Malicious Attack Corporate Awareness and Walk through Date 29 September 2011

1 Figure 4-16: Malicious Software (Malware) Malware: Malicious software Essentially an automated attack robot capable of doing much damage Usually target-of-opportunity

Learning to detect malicious URLs

Beyond Blacklists: Learning to Detect Malicious Web Sites from Suspicious URLs Justin Ma, Lawrence Saul, Stefan Savage, Geoff Voelker Computer Science

Vulnerability Assessment Report - Avenging Security · Clickjacking (User Interface redress attack, UI redress attack, UI redressing) is a malicious technique of tricking a Web user

Introduction to Malicious Code (Malware) - Chalmers · Introduction to Malicious Code (Malware) ... • But what type of content is safe? Targeted attacks ... • Salami attack

Finding Shilling Attack in Recommender System …ksiresearchorg.ipage.com/seke/seke18paper/seke18paper...malicious attacks, especially shilling attack because of data openness and

Attack Toolkits and Malicious Websites

PROTECTING CONNECTED DEVICES AGAINST …files.informatandm.com/uploads/2017/5/CSA_Group...Service (DDoS) attack in the U.S. dramatically demonstrated the impact of a malicious attack

URLDoc: Learning to Detect Malicious URLs using Online Logistic Regression

Turla Waterhole Attack Investigation - RiskIQ · The Turla Waterhole attack was a malicious reconnaissance campaign that targeted ministry and embassy websites. All of the targeted

DNS Cache Poisoning Attack - OWASP€¦ · - April 2018, a major DNS cache poisoning attack compromised Amazon’s DNS servers, redirecting users to malicious web sites. - November

Learning based Malicious Web Sites Detection Using ...users.eecs.northwestern.edu/~hlc720/349/HTXPZYQ_poster.pdf · Learning based Malicious Web Sites Detection Using Suspicious URLs!

New Jersey Association of Counties ISSUE 100 - NOVEMBER 2019 · 5. Filter web and email content. Email containing malicious URLs is the most common ransomware attack method. Implement

zap Jenkins plugin 2 - OWASP€¦ · ZAP JENKINS PLUGIN – FEATURES • Manage Sessions (Load or Persist) • Define Context (Name, Include URLs and Exclude URLs) • Attack Contexts

Malicious Control System Cyber Security Attack Case Study ... · Malicious Control System Cyber Security Attack Case Study– Maroochy Water Services, Australia. 1. Marshall Abrams,

Malicious Accounts Detection based on Short URLs in Twitter