detecting headless browsers

Headless Browser Hide & Seek

Sergey Shekyan, Bei Zhang Shape Security

Who We Are• Bei Zhang

Senior Software Engineer at Shape Security, focused on analysis and countermeasures of automated web attacks. Previously, he worked at the Chrome team at Google with a focus on the Chrome Apps API. His interests include web security, source code analysis, and algorithms.

• Sergey Shekyan

Principal Engineer at Shape Security, focused on the development of the new generation web security product. Prior to Shape Security, he spent 4 years at Qualys developing their on demand web application vulnerability scanning service. Sergey presented research at security conferences around the world, covering various information security topics.

Is There A Problem?

What Is a Headless Browser and How it Works

Scriptable browser environment that doesn’t require GUI

• Existing browser layout engine with bells and whistles (PhantomJS - WebKit, SlimerJS - Gecko, TrifleJS - Trident)

• Custom software that models a browser (ZombieJS, HtmlUnit)

• Selenium (WebDriver API)

What Is a Headless Browser and How it Works

Discussion will focus on PhantomJS:

• Backed by WebKit engine

• Cross-platform

• Popular

• True headless

PhantomJS World

PhantomJSJavaScript Context QWebFrame

QtWebKit

Web Page JavaScript

Context

Control

CallbackInjection

PageEvent

Callbacks areserialized

var page = require('webpage').create();page.open(url, function(status) { var title = page.evaluate(function() { return document.title; }); console.log('Page title is ' + title);});

Legitimate uses and how you can benefit

• Web Application functional and performance testing

• Crawler that can provide certain amount of interaction to reveal web application topology, Automated DOM XSS, CSRF detection

• SEO (render dynamic web page into static HTML to feed to search engines)

• Reporting, image generation

Malicious Use of Headless Browser• Fuzzing

• Botnet

• Content scraping

• Login brute force attacks

• Click fraud

• Bidding wars

Web admins tend to block PhantomJS in production, so pretending to be a real browser is healthy choice

How It Is Different From a Real Browser• Outdated WebKit engine (close to Safari 5 engine, 4 y.o.)

• Uses Qt Framework’s QtWebKit wrapper around WebKit

• Qt rendering engine

• Qt network stack, SSL implementation

• Qt Cookie Jar, that doesn’t support RFC 2965

• No Media Hardware support (no video and audio)

• Exposes window.callPhantom and window._phantom

• No sandboxing

Good vs. Bad

Headless Browser Seek• Look at user agent string

if (/PhantomJS/.test(window.navigator.userAgent)) { console.log(‘PhantomJS environment detected.’);}

Headless Browser Hide• Making user-agent (and navigator.userAgent) a

“legitimate” one:

var page = require(‘webpage').create();

page.settings.userAgent = ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0';

Score board

Phantom Web site

User Agent String Win Lose

Headless Browser Seek• Sniff for PluginArray content

if (!(navigator.plugins instanceof PluginArray) || navigator.plugins.length == 0) {    console.log("PhantomJS environment detected.");  } else {    console.log("PhantomJS environment not detected.");  }

Headless Browser Hide• Fake navigator object, populate PluginArray with whatever values you need.

• Spoofing Plugin objects inside the PluginArray is tedious and hard.

• Websites can actually create a plugin to test it.

• CONCLUSION: Not a good idea to spoof plugins.

page.onInitialized = function () {    page.evaluate(function () {        var oldNavigator = navigator;        var oldPlugins = oldNavigator.plugins;        var plugins = {};        plugins.length = 1;        plugins.__proto__ = oldPlugins.__proto__;

        window.navigator = {plugins: plugins};        window.navigator.__proto__ = oldNavigator.__proto__;    });};

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Headless Browser Seek• Alert/prompt/confirm popup suppression timing

detection

var start = Date.now();  prompt('I`m just kidding');  var elapse = Date.now() - start;  if (elapse < 15) {    console.log("PhantomJS environment detected. #1");  } else {    console.log("PhantomJS environment not detected.");  }

Headless Browser Hide• Can’t use setTimeout, but blocking the callback by

all means would work

page.onAlert = page.onConfirm = page.onPrompt = function () {    for (var i = 0; i < 1e8; i++) {    }    return "a";};

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

Headless Browser Seek• Default order of headers is consistently different

in PhantomJS. Camel case in some header values is also a good point to look at.

PhantomJS 1.9.7GET / HTTP/1.1User-Agent: Accept:Connection: Keep-AliveAccept-Encoding:Accept-Language:Host:

Chrome 37GET / HTTP/1.1Host:Connection: keep-aliveAccept:User-Agent:Accept-Encoding: Accept-Language:

Headless Browser Hide• A custom proxy server in front of PhantomJS

instance that makes headers look consistent with user agent string

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

Headless Browser Seek• PhantomJS exposes APIs:

• window.callPhantom

• window._phantom // not documented.

if (window.callPhantom || window._phantom) {  console.log("PhantomJS environment detected.");} else {  console.log("PhantomJS environment not detected.");}

Headless Browser Hide• store references to original callPhantom, _phantom

• delete window.callPhantom, window._phantom

page.onInitialized = function () {    page.evaluate(function () {        var p = window.callPhantom;        delete window._phantom;        delete window.callPhantom;        Object.defineProperty(window, "myCallPhantom", {            get: function () { return p;},            set: function () {}, enumerable: false});        setTimeout(function () { window.myCallPhantom();}, 1000);    });};page.onCallback = function (obj) { console.log(‘profit!'); };

Unguessable name

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

window.callPhantom Win Lose

Headless Browser Seek• Spoofing DOM API properties of real browsers:

• WebAudio

• WebRTC

• WebSocket

• Device APIs

• FileAPI

• WebGL

• CSS3 - not observable. Defeats printing.

• Our research on WebSockets: http://goo.gl/degwTr

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

window.callPhantom Win Lose

HTML5 features Lose Win

Headless Browser Seek• Significant difference in JavaScript Engine: bind() is

not defined in PhantomJS prior to version 2(function () {    if (!Function.prototype.bind) {      console.log("PhantomJS environment detected.");      return;    }    console.log("PhantomJS environment not detected.");  })();

Function.prototype.bind = function () { var func = this; var self = arguments[0]; var rest = [].slice.call(arguments, 1); return function () { var args = [].slice.call(arguments, 0); return func.apply(self, rest.concat(args)); };};

Headless Browser Seek• Detecting spoofed Function.prototype.bind for

PhantomJS prior to version 2

(function () {    if (!Function.prototype.bind) {      console.log("PhantomJS environment detected. #1");      return;    }    if (Function.prototype.bind.toString().replace(/bind/g, 'Error') != Error.toString()) {      console.log("PhantomJS environment detected. #2");      return;    }    console.log("PhantomJS environment not detected.");  })();

Headless Browser Hide• Spoofing Function.prototype.toString

function functionToString() {    if (this === bind) {        return nativeFunctionString;    }    return oldCall.call(oldToString, this);}

Headless Browser Seek• Detecting spoofed Function.prototype.bind for

PhantomJS prior to version 2(function () {    if (!Function.prototype.bind) {      console.log("PhantomJS environment detected. #1");      return;    }    if (Function.prototype.bind.toString().replace(/bind/g, 'Error') != Error.toString()) {      console.log("PhantomJS environment detected. #2");      return;    }    if (Function.prototype.toString.toString().replace(/toString/g, 'Error') != Error.toString()) {      console.log("PhantomJS environment detected. #3");      return;    }    console.log("PhantomJS environment not detected.");  })();

Headless Browser Hide• Spoofing Function.prototype.toString.toStringfunction functionToString() {    if (this === bind) {        return nativeFunctionString;    }    if (this === functionToString) {        return nativeToStringFunctionString;    }    if (this === call) {        return nativeCallFunctionString;    }    if (this === apply) {        return nativeApplyFunctionString;    }    var idx = indexOfArray(bound, this);    if (idx >= 0) {        return nativeBoundFunctionString;    }    return oldCall.call(oldToString, this);}

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

window.callPhantom Win Lose

HTML5 features Lose Win

Function.prototype.bind Win Lose

Headless Browser Seek• PhantomJS2 is very EcmaScript5 spec compliant,

so checking for outstanding JavaScript features is not going to work

Headless Browser Seek• Stack trace generated by PhantomJs

• Stack trace generated by SlimerJS

• Hmm, there is something common…

at querySelectorAll (phantomjs://webpage.evaluate():9:10)at phantomjs://webpage.evaluate():19:30at phantomjs://webpage.evaluate():20:7at global code (phantomjs://webpage.evaluate():20:13)at evaluateJavaScript ([native code])at global code (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14)

Element.prototype.querySelectorAll@phantomjs://webpage.evaluate():9@phantomjs://webpage.evaluate():19@phantomjs://webpage.evaluate():20

Headless Browser Seek

var err;try { null[0]();} catch (e) { err = e;}if (indexOfString(err.stack, 'phantomjs') > -1) { console.log("PhantomJS environment detected.");} else {  console.log("PhantomJS environment is not detected.");

It is not possible to override the thrown TypeError

generic indexOf(), can be spoofed, define your own

Headless Browser Hide• Modifying webpage.cpp:QVariant WebPage::evaluateJavaScript(const QString &code){ QVariant evalResult; QString function = "(" + code + ")()"; evalResult = m_currentFrame->evaluateJavaScript(function, QString("phantomjs://webpage.evaluate()"));return evalResult;}

at at at at at evaluateJavaScript ([native code])at at global code (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14)

• Produces

Headless Browser Hide• Spoofed PhantomJs vs Chrome:

at querySelectorAll (Object.InjectedScript:9:10) at Object.InjectedScript:19:30 at Object.InjectedScript:20:7 at global code (Object.InjectedScript:20:13) at evaluateJavaScript ([native code]) at at global code (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14)

TypeError: Cannot read property '0' of null at HTMLDocument.Document.querySelectorAll.Element.querySelectorAll [as querySelectorAll] (<anonymous>:21:5) at <anonymous>:2:10 at Object.InjectedScript._evaluateOn (<anonymous>:730:39) at Object.InjectedScript._evaluateAndWrap (<anonymous>:669:52) at Object.InjectedScript.evaluate (<anonymous>:581:21)

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

window.callPhantom Win Lose

HTML5 features Lose Win

Function.prototype.bind Win Lose

Stack trace Lose Win

Score board

Phantom Web site

User Agent String Win Lose

Inspect PluginArray Lose Win

Timed alert() Lose Win

HTTP Header order Win Lose

window.callPhantom Win Lose

HTML5 features Lose Win

Function.prototype.bind Win Lose

Stack trace Lose Win

SCORE: 4 4

Attacking PhantomJS

How to turn a headless browser against the attacker

The usual picture:

• PhantomJS running as root

• No sandboxing in PhantomJS

• Blindly executing untrusted JavaScript

• outdated third party libs (libpng, libxml, etc.)

Can possibly lead to:

• abuse of PhantomJS

• abuse of OS running PhantomJS

Headless Browser Seekvar html = document.querySelectorAll('html');var oldQSA = document.querySelectorAll;Document.prototype.querySelectorAll = Element.prototype.querySelectorAll = function () { var err; try { null[0](); } catch (e) { err = e; } if (indexOfString(err.stack, 'phantomjs') > -1) { return html; } else { return oldQSA.apply(this, arguments); }};

It is not possible to override the thrown TypeError

generic indexOf(), can be spoofed, define your

Headless Browser Seek• In a lot of cases --web-security=false is used in

PhantomJS

var xhr = new XMLHttpRequest(); xhr.open('GET', 'file:/etc/hosts', false); xhr.onload = function () { console.log(xhr.responseText); }; xhr.onerror = function (e) { console.log('Error: ' + JSON.stringify(e)); }; xhr.send();

Headless Browser Seek• Obfuscate, randomize the output, randomize the modified API

call

var _0x34c7=["\x68\x74\x6D\x6C","\x71\x75\x65\x72\x79\x53\x65\x6C\x65\x63\x74\x6F\x72\x41\x6C\x6C","\x70\x72\x6F\x74\x6F\x74\x79\x70\x65","\x73\x74\x61\x63\x6B","\x70\x68\x61\x6E\x74\x6F\x6D\x6A\x73","\x61\x70\x70\x6C\x79"];var html=document[_0x34c7[1]](_0x34c7[0]);var apsdk=document[_0x34c7[1]];Document[_0x34c7[2]][_0x34c7[1]]=Element[_0x34c7[2]][_0x34c7[1]]=function (){var _0xad6dx3;try{null[0]();} catch(e){_0xad6dx3=e;} ;if(indexOfString(_0xad6dx3[_0x34c7[3]],_0x34c7[4])>-1){return html;} else {return apsdk[_0x34c7[5]](this,arguments);};};

Tips for Using Headless Browsers Safely• If you don’t need a full blown browser engine, don’t

use it

• Do not run with ‘- -web-security=false’ in production, try not to do that in tests as well.

• Avoid opening arbitrary page from the Internet

• No root or use chroot

• Use child processes to run webpages

• If security is a requirement, use alternatives or on a controlled environment, use Selenium

Tips For Web Admins• It is not easy to unveil a user agent. Make sure you want

to do it.

• Do sniff for headless browsers (some will bail out)

• Combine several detection techniques

• Reject known unwanted user agents (5G Blacklist 2013 is a good start)

• Alter DOM API if headless browser is detected

• DoS

• Pwn

Start Detecting Already

Ready to use examples:

github.com/ikarienator/phantomjs_hide_and_seek

References• http://www.sba-research.org/wp-content/uploads/publications/jsfingerprinting.pdf

• https://kangax.github.io/compat-table/es5/

• https://media.blackhat.com/us-13/us-13-Grossman-Million-Browser-Botnet.pdf

• http://ariya.ofilabs.com/2011/10/detecting-browser-sniffing-2.html

• http://www.darkreading.com/attacks-breaches/ddos-attack-used-headless-browsers-in-150-hour-siege/d/d-id/1140696?

• http://vamsoft.com/downloads/articles/vamsoft-headless-browsers-in-forum-spam.pdf

• http://blog.spiderlabs.com/2013/02/server-site-xss-attack-detection-with-modsecurity-and-phantomjs.html

• http://googleprojectzero.blogspot.com/2014/07/pwn4fun-spring-2014-safari-part-i_24.html

Thank you!@ikarienator

@sshekyan