a study of static analysis tools for ethereum smart contracts
TRANSCRIPT
A Study of Static Analysis Tools for Ethereum Smart
Contracts
Antonio Pedro Cruz Monteiro
Thesis to obtain the Master of Science Degree in
Information Systems and Computer Engineering
Supervisor: Prof. Joao Fernando Peixoto Ferreira
Examination Committee
Chairperson: Prof. Joao Antonio Madeiras PereiraSupervisor: Prof. Joao Fernando Peixoto Ferreira
Member of the Committee: Prof. Rui Filipe Lima Maranhao de Abreu
October 2019
Acknowledgments
I would like to thank my supervisor Joao F. Ferreira for presenting the theme and for all the support
during this work. I would also like to thank Rui Maranhao and Thomas Durieux for contributions to
SmartBugs and to the research paper that supports Chapter 4. Finally, I thank my family and friends
for the continuous support in all my academic journey.
Abstract
Blockchain technology has been receiving considerable attention from industry and academia, for it
promises to disrupt the digital online world by enabling a democratic, open, and scalable digital economy
based on decentralized distributed consensus without the intervention of third-party trusted authorities.
Smart contracts, computer programs executed on top of a blockchain, are at the core of this technology,
for they allow the creation of new distributed applications. Millions of smart contracts have been deployed
on Ethereum, a major blockchain platform for smart contracts, and although they are seen with great
potential, smart contracts have been known to have several security problems. Recent attacks on smart
contracts and critical vulnerabilities discovered show us the huge financial loss they can impose on the
users and warn us for the necessity of having methods that lead to secure and reliable implementations.
Over the last few years several static analysis tools have been developed specifically targeting Ethereum
smart contracts. In this dissertation, we present a review of state-of-the-art static analysis tools and
introduce SmartBugs, a new extendable execution framework, created to facilitate the integration and
comparison between multiple analysis tools and the analysis of Ethereum smart contracts. SmartBugs
includes two datasets with a total of 47,661 Solidity smart contracts. We use SmartBugs to perform an
empirical evaluation of 7 state-of-the-art automated analysis tools using the two new datasets. Our study
shows that Mythril is the most sensitive and Slither has the best precision. In addition, we present an
extension that improves SmartCheck.
Keywords
Blockchain, Ethereum, Smart Contracts, Static Analysis, Vulnerabilities, Solidity
iii
Resumo
A tecnologia blockchain tem recebido consideravel atencao por parte da industria e academia, pelas suas
promessas em mudar o mundo digital permitindo uma economia digital aberta, democratica e escalavel
baseada em consenso distribuıdo de forma descentralizada. Smart contracts, programas de computador
executados em blockchain, sao parte nuclear desta tecnologia, pois permitem a criacao de novas aplicacoes
distribuıdas. Milhoes de smart contracts foram submetidos na rede Ethereum, a maior plataforma de
blockchain para smart contracts e, embora sejam vistos com grande potencial, sabe-se que smart contracts
tem tido varios problemas de seguranca. Ataques recentes a smart contracts e vulnerabilidades crıticas
descobertas mostram a enorme perda financeira que podem causar aos utilizadores e alertam-nos para a
necessidade de uma implementacao segura e confiavel, alertando para a necessidade de aplicar tecnicas
de verificacao. Nos ultimos anos, foram desenvolvidas varias ferramentas de analise estatica visando
especificamente smart contracts da rede Ethereum. Nesta dissertacao apresentamos uma revisao de
ferramentas de analise a smart contracts e introduzimos o SmartBugs, um novo framework de execucao
extensıvel, criado para facilitar a integracao e a comparacao entre varias ferramentas de analise e a analise
de smart contracts. O SmartBugs inclui dois datasets com um total de 47,661 smart contracts escritos
em Solidity e e utilizado para realizar uma avaliacao empırica de 7 ferramentas de analise estatica usando
os dois novos datasets. O nosso estudo mostra que Mythril e a ferramenta mais sensıvel e Slither tem a
melhor precisao. Alem disso, apresentamos uma extensao que melhora o SmartCheck.
Palavras Chave
Blockchain, Ethereum, Smart Contracts, Analise Estatica, Vulnerabilidades, Solidity
v
Contents
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Topic Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Background 7
2.1 Blockchain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Ethereum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Vulnerabilities of Ethereum Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.1 Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.2 DASP 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4.2.1 Reentrancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4.2.2 Access control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.2.3 Arithmetic Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.2.4 Unchecked Low Level Calls . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.2.5 Denial of Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.2.6 Bad Randomness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.2.7 Front Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.2.8 Time manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.2.9 Short Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.2.10 Unknown Unknowns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Static Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 State Of The Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 SmartBugs: An Execution Framework 31
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
vii
3.2 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.1 sbcurated: A Dataset of 143 Vulnerable Smart Contracts . . . . . . . . . . . . . . . 35
3.3.1.1 sbcurated Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4 Framework Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.2 Methodology for adding tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4.3 Methodology for filtering bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.4 CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.5 WUI Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4 Analysis: An Empirical Review 43
4.1 Analysis of 47,587 Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1.1 Data Collection: sbwild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1.2 Tools’ Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1.3 Large-scale Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1.4 sbcurated: Analysis of 69 Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . 46
4.1.5 sbwild: Vulnerabilities in Production Smart Contracts . . . . . . . . . . . . . . . . 49
4.1.6 Execution Time of the Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2 Precision of the Static Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.1 Testing Environment and Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . 53
4.2.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5 SmartCheck Extension 55
5.1 SmartCheck Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 Patterns Added . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4 Availabity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6 Conclusion 63
6.1 Achievements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
viii
List of Figures
2.1 Chain of blocks (Nakamoto, 2008) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Schematic of a blockchain platform with smart contracts. (Delmolino et al., 2015) . . . . 11
2.3 Example execution of a smart contract on Ethereum. (Fontein, 2018) . . . . . . . . . . . . 12
2.4 Overview of workflow in the Ethereum network (Bhargavan et al., 2016) . . . . . . . . . . 13
2.5 The recursive loop of a reentrancy attack. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1 SmartBugs Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 SmartBugs Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3 Input panel for smart contracts and tool’s choice (Oyente, Slither and SmartCheck are
selected) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4 Number of issues found by each tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.5 Issues found by Oyente . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.1 Proportion of vulnerabilities identified by exactly one (1), two (2) or three (3) tools, and
by four tools or more (4+) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2 Evolution of number of vulnerabilities over time. . . . . . . . . . . . . . . . . . . . . . . . 51
4.3 Correlation between the number of vulnerabilities and balance in Wei (one Ether is 1018
Wei) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.1 Parse tree for the Balance equality code example (SmartCheck, 2018) . . . . . . . . . . . 57
ix
x
List of Tables
2.1 Taxonomy of vulnerabilities by Atzei et al. [1] . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 DASP Top 10 vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Solidity methods to transfer Ether . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Tools identified and their URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5 Excluded analysis tools based on our inclusion criteria. . . . . . . . . . . . . . . . . . . . . 29
2.6 Included analysis tools based on our inclusion criteria. . . . . . . . . . . . . . . . . . . . . 29
3.1 Categories of vulnerabilities available in the dataset. (computed using cloc 1.82). . . . . . 36
4.1 Statistics on the collection of Solidity smart contracts from the Ethereum blockchain. . . 46
4.2 Categories of vulnerabilities available in the dataset sbcurated used in the study. (computed
using cloc 1.82). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Vulnerabilities identified per category by each tool. . . . . . . . . . . . . . . . . . . . . . . 48
4.4 Total number of detected vulnerabilities by each tool, including vulnerabilities not tagged
in the dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.5 Combined tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.6 The total number of contracts that have at least one vulnerability (analysis of 47,518
contracts). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.7 Average execution time for each tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.8 Metrics of the tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.1 Vulnerabilities identified per category by SmartCheck and SmartCheck Extended in sbcurated. 60
5.2 SmartCheck and SmartCheck Extended metrics. . . . . . . . . . . . . . . . . . . . . . . . 61
5.3 Positives of SmartCheck Extended. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
xi
xii
Listings
2.1 Vulnerable - accounts is updated after message call. . . . . . . . . . . . . . . . . . . . . . 16
2.2 Safe - accounts is updated before message call. . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Simplified example of The DAO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Malicious contract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5 Modifier defined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.6 Vulnerable - changeOwner is not protected. . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7 Safe - modifier is used correctly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.8 Vulnerable - does not check for overflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.9 Safe - checks for overflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.10 Safe - using SafeMath library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.11 Vulnerable - send() can fail and prizePaidOut will be sent to true regardless. . . . . . . . 21
2.12 Safe - prizePaidOut will be sent to true if send() is true. . . . . . . . . . . . . . . . . . . . 22
2.13 Vulnerable - send may fail and no one else will be able to bid. . . . . . . . . . . . . . . . . 22
2.14 Safe - the refund is recorded safely. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.15 Vulnerable to bad randomness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.16 Vulnerable to Front Running. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.17 Vulnerable to time manipulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1 Balance equality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 XPath pattern for the Balance equality issue. . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3 SOLIDITY BAD RANDOMNESS : XPath pattern to detect bad randomness vulnerabilities. 58
5.4 SOLIDITY UNPROTECTED : XPath pattern to detect unprotected ownership transfers. 59
5.5 SOLIDITY UNPROTECTED : XPath pattern to detect unprotected suicides. . . . . . . . 59
5.6 SOLIDITY EXACT TIME : XPath pattern to detect time manipulation vulnerabilities. . 60
xiii
xiv
1Introduction
Contents
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Topic Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1
2
1.1 Motivation
Blockchain technology and cryptocurrencies, introduced by Bitcoin [2], have gained and continue to gain
attention and popularity. Blockchain, a shared, decentralized, cryptographically secure and immutable
digital ledger maintained by nodes in a peer-to-peer network is considered to be one of the most prominent
technologies introduced in this century. Cryptocurrencies, which have become a global phenomenon, use
this data structure as a public ledger to record all valid transactions. Blockchain acts as public append-
only database that keeps a permanent and immutable record of digital transactions without relying on
trust to secure it, instead it is secured by a peer-to-peer network using a consensus algorithm.
One of the blockchain appeals is being a technology that promises secure distributed computations
even in the absence of trusted third parties. These computations, called smart contracts — interactions
between mutually distrusting entities, are automatically enforced by the consensus mechanism of the
blockchain, using incentives and cryptography. These immutable pieces of code, when deployed on the
blockchain, allow parties to manifest contract terms in the form of program code.
Most transactions made today are by means of intermediaries. These trusted third parties facilitate
transactions and provide the connection between the entities involved, overcoming trust issues and en-
suring that the agreement is fulfilled. Smart contracts can present themselves as substitutes for these
third parties as they can overcome the issue of missing trust. Multiple blockchain platforms have been
presented since the introduction of Bitcoin, extending the number of applications of the technology.
Ethereum [3], which was introduced with the proclaimed intention of being a platform aimed to support
smart contracts, gave birth to a new generation of the blockchain — the second generation. Ethereum
is the most famous network to deploy smart contracts, where they are written in a Turing-complete
language. Solidity is the most used language by developers to create smart contracts on Ethereum.
While smart contracts can represent a great improvement to our society model, as they do not rely
on gained trust, they have shown several major security vulnerabilities. In a study performed on nearly
one million Ethereum smart contracts, 34,200 of them were flagged as vulnerable [4]. A different study
showed that of 19,366 smart contracts analysed, also in Ethereum, 8,833 (around 46%) were flagged as
vulnerable [5]. A need to review and improve smart contracts implementation is therefore needed and
it is a key element to enable massive adoption of smart contracts technology. Recent incidents related
to bugs on smart contracts have been linked with losses of millions in cryptocurrency (see Section 2.4),
warning us for the urgent need of safer smart contracts. We propose the use of static analysis tools as
part of the solution, helping finding potential mistakes and preventing unsafe code to be deployed on the
blockchain.
3
1.2 Objectives and Contributions
We propose to explore and discuss smart contracts and their secure implementation using static analysis
tools, which can be used to analyse the correctness of the code. The main objectives are to study analysers
that verify Ethereum smart contracts and introduce new tools and datasets to facilitate the use of static
analysis in research and smart contracts development.
To achieve these objectives, we start by providing an overview on blockchain technology to later focus
on Ethereum smart contracts. We overview the top 10 vulnerabilities categories found in Ethereum smart
contracts, discuss major attacks and their mitigation, and collect a list of state-of-the-art automated smart
contract analysers that analyse Ethereum smart contracts.
Later, we describe a novel, extendable, and easy-to-use execution framework, SmartBugs, that simpli-
fies research and execution of automated analysis tools for smart contracts. With this framework we also
introduce a dataset of 143 annotated vulnerable Solidity smart contracts, sbcurated, setting the ground
for fair comparisons and a reference for the research community to use for benchmarks on analyses tools.
We use SmartBugs to execute 7 state-of-the-art static analysis tools on two datasets, sbcurated and
sbwild, with a total of 47,587 contracts being analysed. The second dataset contains all available smart
contracts in the Ethereum blockchain that have Solidity source code available on Etherscan1 (a total
of 47,518 contracts). We perform an empirical analysis of this 7 tools on this datasets and present a
discussion of the results.
To conclude, we present an extension of SmartCheck, a state-of-the-art static analysis tool, improving
its detection of vulnerabilities and adding the capability to detect a category of vulnerabilities that
previous was not in the scope of detection of SmartCheck.
Throughtout this work, we propose to answer the following research questions:
1. What are the state-of-the-art analysis tools for Ethereum smart contracts?
2. What is the precision of the state-of-the-art analysis tools in detecting vulnerabilities on Ethereum
smart contracts?
3. Which categories of vulnerabilities are most detected?
4. How many vulnerable contracts are present in the Ethereum blockchain?
5. How long do the tools require to analyse the smart contracts?
6. How can we improve a state-of-the-art analysis tool?
1Etherscan: https://etherscan.io
4
Our main contributions can be summarized as follows:
• We review, explain and discuss vulnerabilities found on Ethereum smart contracts;
• We present an overview of state-of-the-art automated smart contract analysers focused on Ethereum
smart contracts;
• We introduce SmartBugs, an extendable execution framework designed to facilitate the integration
and comparison between multiple analysis tools and the analysis of Ethereum smart contracts;
• We present sbcurated, a dataset of 143 annotated vulnerable Solidity smart contracts with a defined
vulnerability taxonomy as a reference dataset to the research community;
• We provide an analysis of the execution of 7 state-of-the-art analysis tools with SmartBugs on
47,587 smart contracts;
• We extend SmartCheck, a state-of-the-art static analysis tool, to improve detection of vulnerabili-
ties.
1.3 Topic Overview
Smart contract programming is in many ways similar to traditional programming, but raises important
new security challenges and has unique characteristics. Virtual currencies have real value and smart
contracts are directly related to them. If a smart contract has a vulnerability it can represent a potential
financial loss.
Recently a set of static analysis tools have been developed and introduced specifically targeting
Ethereum smart contracts. We aim to study static analysis tools for Ethereum smart contracts while
introducing an execution framework to facilitate their execution and thus set an easier environment to
execute static analysis on smart contracts.
In smart contracts security should be of paramount importance for multiple reasons [6]:
• Recent technology - The concept of smart contracts is relatively new and their execution envi-
ronment differs from the most popular: mobile, desktop or cloud. Developers are not used to their
code being executed in a blockchain, a decentralzized peer-to-peer network.
• Immutability - Once deployed, a smart contract can not be patched or corrected (although some
workarounds exists, e.g using a proxy pattern). The correctness of the contract must thus be verified
before deployment.
• Motivated hackers - Due to the financial value of smart contracts, hackers are more motivated to
exploit vulnerabilities. Recent history show us the huge financial loss imposed by attacks on smart
contracts.
5
• Rapid pace of development - Blockchain industry and development is growing fast, smart con-
tracts programming and use are increasing quickly. Also, smart contracts programming languages
are quickly evolving. For example, Solidity, the main programming language used to write smart
contracts on Ethereum, has a very fast pace and suffers updates constantly.
1.4 Thesis Outline
The remainder of the document is structured as follows. Chapter 2 overviews blockchain technology,
the Ethereum network, smart contracts, and presents a review of smart contract vulnerabilities and
static analysis tools. Chapter 3 introduces SmartBugs, an extendable execution framework, and the
dataset sbcurated. Chapter 4 provides an empirical review of the analysis of two datasets, sbcurated and
sbwild, using SmartBugs. The first section of Chapter 4 was co-authored by three INESC-ID researchers,
submitted to ICSE 2020 and published in [7]. Chapter 5 describes a proposal to improve SmartCheck’s
capability to detect vulnerabilities. Finally, Chapter 6 presents conclusions regarding the work done and
discusses future work.
6
2Background
Contents
2.1 Blockchain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Ethereum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Vulnerabilities of Ethereum Smart Contracts . . . . . . . . . . . . . . . . . 14
2.5 Static Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
7
8
In this chapter we start by presenting an overview of Blockchain and smart contracts technology.
Later we focus on Ethereum and discuss security issues within Ethereum smart contracts and tools to
perform static analysis.
This chapter is divided in five sections: Blockchain, Smart Contracts, Ethereum, Vulnerabilities of
Ethereum Smart Contracts and Static Analysis Tools.
2.1 Blockchain
In 2008 Satashi Nakamoto proposed Bitcoin, “a system for electronic transactions without relying on
trust” [2], presenting a decentralized peer-to-peer network using proof-of-work, a consensus protocol, as
a solution to the double-spending1 problem. In this scenario, bitcoin (BTC) represents a cryptographic
currency that can be exchanged between untrusted parties and the key idea is that the exchange of
bitcoins between these entities is registered in a append-only database maintained by a network of peers
that can not be tampered. This append-only database is built by chaining multiple validated blocks that
contain transactions and is referred as blockchain. Blockchains can be divided in two different types:
permissionless and permissioned. Bitcoin is the first permissionless blockchain, meaning that everyone
can participate and interact with the network. Permissioned blockchains, on the other hand, are more
restricted and the participants are controlled.
Blockchain can be described as a continuously ordered list of records, which are bundled in blocks
and can consist of any information - cryptocurrency, computer code or any other kind of data. These
blocks are linked one to another, like a chain, where each block references the block that came before
it and is identified by its cryptographic hash, as shown in Figure 2.1, making the links between blocks
virtually unbreakable. Each block consists of a set of transactions and once created and appended to the
blockchain, the transactions in that block become immutable, cannot be changed or reverted, ensuring
the integrity of the transactions. This data structure is shared by the nodes in the network. For a record
to be appended in the blockchain it needs to be verified and validated.
Figure 2.1: Chain of blocks (Nakamoto, 2008)
Any user can create and interact with permissionless blockchain by publishing a signed transaction.
These transactions are broadcasted to the network. It is then executed by the nodes in the network (min-
1Double-spending is the result of successfully spending the same digital token more than once. It is a potential flaw ina digital cash systems in which the same single digital token can be spent more than once.
9
ers), where the correct execution is ensured by the consensus protocol. The miners propagate information,
store data, and when a consensus is reached they update the state on the blockchain accordingly.
The consensus protocol enables mutually distrusting parties to create transactions and reach consensus
on the outcome of the execution and is a crucial layer of a blockchain. There are several protocols that
can be used to reach consensus. Commonly, in permissionless blockchains, it consists either in proof
of work (Bitcoin and Ethereum at the time of writing) or proof of stake. In Proof of work (PoW)
miners need to compute and solve an exceptionally difficult computational problem but easily verifiable
to be able to validate a block and get their reward. Finding the solution to this difficult computational
problem is an arduous guessing game and requires a considerable amount of computing power to compete
with all the miners in the network to produce the correct output. Once a miner computes the correct
answer, the block is broadcasted to the network for the other miners to verify it. The other miners in
the network check if the hash is valid, and if it is, they accept the block from the miner. In contrast,
Proof of stake (PoS) states that validators (miners in PoS) can validate block transactions according to
how many coins they hold. The validator stakes an amount of the available currency to verify a block of
transactions, meaning that the bigger the amount staked, the more validation power the validator has.
The cryptographic calculations in PoS are much simpler to solve than in PoW and thus do not require
as much computational power. In PoS, unlike PoW, there is no coin mining; instead, validators collect
transaction fees instead of newly minted coins.
In terms of costs and energy consumption, proof of stake systems are more cost-efficient and thus more
environmental appealing than proof of work systems, as they do not require the intensive computational
power that proof of work requires. Regarding security, proof of stake can reduce the possibility of
malicious attacks. Carrying out an attack in PoS is expensive, since if a validator acts maliciously their
validation power will be revoked and their amount staked will be ceased. PoS systems can also be seen
as more decentralized and as a more fair solution, since the amount of network control a participant can
gain in a PoS system is directly proportional to the amount staked. In PoW, the more the miner invests
in equipment, the more computational power they will have, making it increasingly less profitable and
more difficult for individuals to compete against large mining pools. In Bitcoin, a disproportionately
large number of blocks are mined by pools rather than by individual miners [8].
2.2 Smart Contracts
So far we have mentioned the concept of smart contracts and briefly explained it. We will now go deeper
into smart contracts implementation and applications.
Smart contracts are pieces of code, deployed on a Blockchain, where it is possible to codify agreements
and trust relations. They are able to keep a record of the state, exchange digital assets, take user input
and store data. But the concept of smart contracts is not new. Actually, it has been around for more
10
than two decades as the term was first defined in 1996 by Nick Szabo.
Szabo describes a smart contract as a “ set of promises, specified in digital form, including protocols
within which the parties perform on the other promises” [9]. In his work, Szabo says that “the primitive
ancestor of smart contracts, is the humble vending machine” [9] as it takes coins, delivers change and
gives a product in return. This process is done in an automated way like a finite state machine. With the
development of blockchain technology, Szabo idea of smart contracts gained a new life and possibility.
Blockchain became the perfect host.
Smart contracts, autonomous applications composed by fields and functions, can now bring to the
real world the idea envisioned by Szabo, being stored and executed in a blockchain. As any software
application driven to produce an expected result, a smart contract should always be deterministic, i.e.
the same input should always produce the same output. That being, what we call now a smart contract
can be said to be a technical implementation of Szabo’s idea: a computer program executed on the
blockchain.
Figure 2.2: Schematic of a blockchain platform with smart contracts. (Delmolino et al., 2015)
Smart contracts are stored and executed by the network of nodes who update the state of the smart
contract on the blockchain when consensus on the outcome of the execution is reached. Users can
exchange value or data and interact with contracts by publishing signed transactions to the network.
A schematic of a blockchain with smart contracts is shown in Figure 2.2. Smart contract execution
sometimes requires heavy computational tasks and since they are performed by computers (nodes) in
the network, each computational task has a cost, which is called gas in Ethereum. These execution fees
represent the cost paid by the user to the miners to execute code.
What makes smart contracts such a powerful concept, needless trust between parties involved and
immutability, can be also for seen as a weakness and must be handled carefully. Any mistake or bug
in the code of a smart contract can be exploited. The code is near impossible to correct (if the smart
contract was not designed to suffer upgrades) so a smart contract with vulnerabilities can not be patched.
11
In blockchain, code is law. A need of proper code verification is thus crucial to make the contract as safe
and reliable as possible, before it is deployed.
2.3 Ethereum
Ethereum [3] was introduced in 2014 and was the first to introduce a Turing-complete language in a
blockchain platform, creating a second generation of blockchains. It has its own cryptocurrency, called
Ether (ETH), and smart contracts represent one of the main components of the platform. Ethereum, like
Bitcoin, is a permissionless blockchain where any network participant is allowed to interact and perform
transactions within the network, such as transfer Ether or interact with a smart contract deployed on
the network.
There are two types of accounts in Ethereum: Externally owned accounts (EOAs) and contract
accounts. EOAs are controlled by private keys that can be used to make transactions and sign data and
have no code associated, similar to bitcoin wallet addresses. The owner of the private key has full control
of the account. Contract accounts are controlled by their contract code. They can receive funds and make
transactions like EOAs, but there is generally no single private key that is used to make transactions.
Instead, the logic on how transactions are made is defined via the smart contract code. Unlike EOAs,
contract accounts can not initiate new transactions. Any transaction in Ethereum is always initiated by
an EOA.
Figure 2.3: Example execution of a smart contract on Ethereum. (Fontein, 2018)
The sequence of transactions determines the state of each contract and the balance of each user. A
transaction uses the state stored in the previous block to compute a new state and both the transaction
and the new state are stored in the new block, as seen in Figure 2.3. These transactions will then be
12
grouped in blocks and appended to the blockchain, using miners - untrusted peers in the network - to
process it. Potential conflicts are resolved using a consensus mechanism, in this case Proof of Work
(Although it is still the current mechanism used in Ethereum, a future implementation called Casper is
scheduled to change the consensus protocol to Proof of Stake [10]).
Figure 2.4: Overview of workflow in the Ethereum network (Bhargavan et al., 2016)
Smart contracts are compiled into a stack-based bytecode language and then executed by the Ethereum
Virtual Machine (EVM). EVM is a turing complete virtual machine2, specific to Ethereum that pro-
cesses every transaction. Every participating node in the network runs their own instance of the EVM
making possible the execution of code in a trustless environment. Once deployed, the smart contract
becomes immutable and each instruction execution will have a consumption cost, a positive amount of
gas - computational budget. A gas limit is specified by the sender of the transaction, which when ex-
ceeded results in an exception that reverts the effects of the current transaction. The Ether price of a
gas unit is determined by the market. To deploy a contract to the network a transaction is broadcasted
containing its bytecode and initialization parameters. Like any other transaction, after being validated it
will be included in a block. The contract will be permanently stored at a unique address, used by users
to interact with the smart contract. Users broadcast transactions to that address defining the function to
be called, and, if necessary, its arguments. Contracts can also interact with other contracts. An overview
of the workflow of the network is presented in Figure 2.4.
Solidity is the most used language by developers to create smart contracts on Ethereum. It is an
object-oriented, high-level JavaScript-like language and is compiled to EVM bytecode to be executed
by the EVM. Solidity is statically typed, supports inheritance, libraries and, like many programming
languages, it offers common features like control flow structures, types, functions or structs. Several
examples of Solidity code are shown in the next section.
2EVM code can encode any computation that can be conceivably carried out, including infinite loops.
13
2.4 Vulnerabilities of Ethereum Smart Contracts
In this section our goal is to overview some of the vulnerabilities found in smart contracts, more specifically
in Ethereum smart contracts, and explain them while presenting exploits, mitigation methods and real
world attack scenarios.
2.4.1 Taxonomies
There are multiple kinds of vulnerabilities and they can be presented in different forms. Atzei et al. [1]
presented a taxonomy of vulnerabilities, regarding Ethereum smart contracts, providing the vulnerability
and the level at which the vulnerability occurs - shown in Table 2.1. We also added a cause to briefly
describe each vulnerability.
Level Vulnerability Cause
Solidity
Call to unknown Called function does not exitGasless send Callee’s fallback function is invoked
Exception disorders Bad exception handlingType casts No expectation is thrown, if type-mismatchedReentrancy Non-recursive function reenter before termination
Keeping secrets Contracts private fields, secrecy is not guaranteed
EVMImmutable bugs Defective contracts cannot be patched
Ether lost in transfer Ether sent to orphan address is lostStack size limit The number of values exceeds 1024 in the stack
BlockchainUnpredictable state State of the contract is changed before invoking
Generating randomness Malicious miner biases the outcomeTime constraints Block timestamp is changed by malicious miner
Table 2.1: Taxonomy of vulnerabilities by Atzei et al. [1]
In our work we opted for a more broad and prevailing approach to explain the vulnerabilities found in
smart contracts, following the taxonomy of DASP Top 10 [11] (shown in Table 2.2). The DASP taxonomy
seems to be more prominent as it is updated when necessary, meaning that vulnerabilities such as stack
size limit are not included, as this has been fixed.
We also added a risk level associated to each vulnerability (1 - low, 2 - medium, 3 - high) based on
our research [6] and attacks that will allow us to later isolate high risk vulnerabilities.
Some vulnerabilities may be referred by different names, as seen throughout our research. To avoid
confusion, in Table 2.2 we associate the ones presented by Atzei et al. [1] to a category presented in the
DASP Top 10 and when describing/explaining them we will provide more than one name. Although the
association can be said to be subjective, as one may argue that one of the vulnerabilities defined by Atzei
et al. can fit in more than one category in DASP Top 10, we decided to do a single association based on
the main focus of the vulnerability.
14
DASP Top 10 Atzei et al. Risk
1 ReentrancyCall to unknown
Reentrancy3
2 Access control - 33 Arithmetic - 1
4 Unchecked low level callsGasless send
Exception disorders3
5 Denial of Service - 36 Bad randomness Generating randomness 27 Front running Unpredictable state 28 Time manipulation Time constraints 29 Short addresses - 110 Unknown unknowns - -
Table 2.2: DASP Top 10 vulnerabilities
Ether lost in transfer, Immutable bugs, Type casts and Keeping secrets are vulnerabilities
defined in Atzei et al. work that do not fit our association. These vulnerabilities are relatively sim-
ple concepts and are already briefly explained in Table 2.1. We will now overview the DASP Top 10
vulnerabilities.
2.4.2 DASP 10
In this subsection we discuss each vulnerability category of the taxonomy previously defined.
2.4.2.1 Reentrancy
It is considered to be one of the most severe vulnerabilities found in smart contracts so far, resulting on
the biggest attack ever made on Ethereum – The DAO [12]. Reentrancy bugs refer to reentrant function
calls which can make a contract to behave in an unexpected way.
Definition 1. Reentrancy “Any interaction from a contract (A) with another contract (B) and any
transfer of Ether, hands over control to that contract (B). This makes it possible for B to call back into
A before this interaction is completed.” [13]
Exploiting this bug, a malicious contract B is able to withdraw multiple times and empty the balance
of a contract A making reentrant calls that take advantage of the intermediate state.
A reentrancy attack is carried out by manipulating the control flow in a way that was not intended by
the smart contract developer. Consider the function withdraw() seen in Listing 2.1 as part of an arbitrary
contract. accounts maps how much Ether is owned by each user. The function withdraw() allows users
to withdraw an amount of Ether. The function checks if the amount is equal or inferior to the caller’s
balance, sends the Ether and then updates the balance of the account. withdraw() is vulnerable to a
reentrant call on line three, where the accounts map as not yet been updated.
15
Listing 2.1: Vulnerable - accounts is updated after message call.
1 function withdraw(uint amount) public {2 require(accounts[msg.sender] >= amount);
3 require(msg.sender.call.value(amount)());
4 accounts[msg.sender] -= amount;5 }
Attackers can implement a malicious contract containing a special fallback function3 to exploit with-
draw(). This fallback function would be triggered by the Ether transfer in line 3 and take over control
flow. It would be then possible to issue a new call of withdraw() in the fallback function, before the
balance has been updated in line 4. This would trigger the malicious fallback function once again. This
iterative process (seen in Figure. 2.5) would continue until a point specified by the attacker or until the
transaction runs out of gas.
Figure 2.5: The recursive loop of a reentrancy attack.
Call to the unknown, described by Atzei et al [1], is related to reentrancy as it also refers to the
side effect of invoking the fallback function of the callee/recipient.
A possible solution to avoid the reentrancy problems is to use Checks-Effects-Interactions: first check
the invariants, then update the internal state, then communicate with external entities (as shown in
Listing 2.2).
Listing 2.2: Safe - accounts is updated before message call.
1 function withdraw(uint amount) public {2 require(accounts[msg.sender] >= amount);
3 accounts[msg.sender] -= amount;
4 require(msg.sender.call.value(amount)());
5 }
3A function with no name and no arguments, which is called in case no function identifier matches the called function.
16
Table 2.3 shows the three methods available in Solidity to send Ether, along with important character-
istics that should be taken into account when developing smart contracts. Avoiding the use of call.value()
whenever possible is also recommended, as addr.call.value(x)() transfers x Ether and forwards all gas to
addr and returns false in case of failure, potentially leading to vulnerabilities like reentrancy. The method
send() should also be used with caution as it also returns false on failure and does not throw an excep-
tion, so the execution, if not carefully checked the return value, continues. Instead the use of transfer
is encouraged to prevent reentrancy vulnerabilities because it only provides the callee with a stipend of
2300 gas and throws an exception in case of failure, stopping the code execution. Introducing a mutex is
also an alternative technique, i.e., adding a state variable which locks the contract during code execution,
preventing reentrancy calls.
Method Edit Gas Limit Gas Limit Return Value on Failureaddress.send() No 2300 False
address.transfer() No 2300 Throws Exceptionaddress.call.value()() Yes Editable False
Table 2.3: Solidity methods to transfer Ether
The DAO was a contract designed to implement a crowd-funding platform, where participants would
receive DAO tokens and then would be able to vote for which projects to fund [12]. It raised more than
$150M before being attacked on June 18th, 2016 (the attacker was able to take control of $60M). Later
a hard-fork of Ethereum network nullified the effects of the hack. To get a better understanding and a
basic notion of how the attack occurred consider the simplified example of The DAO shown in Listing 2.3.
Listing 2.3: Simplified example of The DAO.
1 contract SimpleDAO {2
3 mapping (address => uint) public credit;4
5 function donate(address to) public {6 credit[to] += msg.value;7 }8
9 function withdraw(uint amount) public {10 if (credit[msg.sender]>= amount) {11 msg.sender.call.value(amount)();
12 credit[msg.sender]-=amount;13 }14 }15
16 function queryCredit(address to) public view returns (uint){17 return credit[to];18 }19 }
17
SimpleDAO allows participants to donate ether to fund contracts at their choice. Contracts can then
use withdraw to retrieve their funds. To be able to perform an attack and steal ether from SimpleDAO
we need to publish the malicious contract, we will use Mallory contract as an example - shown below in
Listing 2.4.
Listing 2.4: Malicious contract.
1 contract Mallory {2
3 SimpleDAO public dao;4 address owner;5
6 function Mallory(address simpleDaoAddress) public{7 owner = msg.sender;8 dao = SimpleDAO(simpleDaoAddress);9 }
10
11 function getJackpot() public {12 owner.send(this.balance);13 }14
15 function() external payable {16 dao.withdraw(dao.queryCredit(this));17 }18 }
To go through the attack, an adversary would donate some ether to Mallory which would invoke
Mallory’s fallback function (line 15). This function invokes withdraw, which transfers the ether to Mallory
and invokes again Mallory ’s fallback function. Doing this invoke again, withdraw is maliciously called
back and so the credit was not updated - the check at line 10 succeeds. As a consequence SimpleDAO
sends the credit to Mallory for the second time, invoking again the fallback function. This loop will
continue until the balance of SimpleDAO becomes zero, the call stack is full or until the transaction runs
out of gas. Further there is no gas limit specified in the withdraw function at line 11 so the adversary can
delay the out-of-gas exception by providing more gas in the originating transaction. getJackpot() allows
the attacker to get the funds from Mallory. An attack like this would allow the a malicious user to steal
all ether from the contract SimpleDAO.
2.4.2.2 Access control
Access Control problems are not only related to smart contracts but are common in all programs. In
smart contracts these vulnerabilities can occur when tx.origin is used for authorization, when a contract
handles large authorization logic with lengthy require or makes reckless use of delegatecall (special variant
of a message call, the code at the target address is executed in the context of the calling contract) in
proxy libraries or proxy contracts. Failure to use function modifiers to restrict access allows an attacker
18
to manipulate contracts. All sensitive functions on a smart contracts must be carefully protected.
Definition 2. Access Control “Access control enforces a policy such that users cannot act outside of
their intended permissions. Failures typically lead to unauthorized information disclosure, modification
or destruction of all data, or performing a business function outside of the limits of the user.” [14]
Restricting access is a common and crucial pattern for smart contracts. Restrict the view of the
source code of a smart contract, transactions or contract’s state is not feasible on a blockchain. Restrict
modifications to a contract’s state or functions on the other hand is useful and of paramount importance.
Modifiers are an easy and readable way of doing it.
“Modifiers can be used to easily change the behaviour of functions. For example, they can
automatically check a condition prior to executing the function. Modifiers are inheritable
properties of contracts and may be overridden by derived contracts.” - as explained in [13]
In Listing 2.5 a modifier is defined. The function body is inserted where the special symbol “ ;” in
the definition of a modifier appears. This means that if the owner calls this function, the function is
executed and otherwise, an exception is thrown and the code execution is stopped.
Listing 2.5: Modifier defined.
1 modifier onlyOwner {2 require(msg.sender == owner);3 ;4 }
Now imagine that we want to change the owner of that contract, creating a changeOwner method.
If changeOwner does not use the modifier and the access is not restricted to the current owner it allows
anyone to become the contract owner, opening a possible attack window - as seen in Listing 2.6. To
correctly change the owner we should use the modifier previously defined, as shown in Listing 2.7
Listing 2.6: Vulnerable - changeOwner is not protected.
1 function changeOwner(address newOwner) public {2 owner = newOwner;3 }
Listing 2.7: Safe - modifier is used correctly.
1 function changeOwner(address newOwner) public onlyOwner {2 owner = newOwner;3 }
19
In conclusion a modifier should always be specified for functions to restrict access. Also tx.origin
should never be used to validate callers, instead the use of msg.sender is recommended as tx.origin refers
to the origin of the transaction, the user who originally sent the transaction, and msg.sender to the
immediate sender.
The two Parity wallet hacks [15] and Rubixi are real world examples of this vulnerability.
2.4.2.3 Arithmetic Issues
Usually referred as integer over/underflows, arithmetic issues are not new types of vulnerabilities and are
common in many programming languages.
Definition 3. Arithmetic “An over/under flow occurs when an operation is performed that requires a
fixed size variable to store a number (or piece of data) that is outside the range of the variable’s data
type.” [16]
Looking at Listings 2.8 and 2.9 we can see a simple example where an overflow may occur and how
to mitigate it, respectively. The developer must always be aware of data manipulation and decide if a
over/underflow check is necessary. Using a trusted mathematical libraries, as SafeMath, is also a possible
solution, as seen in Listing 2.10. Using trusted mathematical libraries is encouraged as it abstracts the
developer from validating all arithmetic operations by simply using the library to perform arithmetic
operations.
Listing 2.8: Vulnerable - does not check for overflows.
1 function add(uint value) public returns (bool) {2 sellerBalance += value;3 }
Listing 2.9: Safe - checks for overflows
1 function safe add(uint value) returns (bool) {2 require(value + sellerBalance >= sellerBalance);
3 sellerBalance += value;4 }
20
Listing 2.10: Safe - using SafeMath library.
1 import "./SafeMath.sol"
2
3 contract SafeArithmeticContract {4
5 using SafeMath for uint256;
6 uint256 result;7
8 function SafeAdd(uint256 a, uint256 b) {9 result = 0;
10 result = a.add(b);11 }12 }
Validation of all arithmetic data and the use of trusted mathematical libraries, as SafeMath library,
are thus encouraged as a preventive method. BatchOverflow and ProxyOverflow show us the real impact
arithmetic issues can have, in particular integer overflow.
2.4.2.4 Unchecked Low Level Calls
Related to the use of low level functions, as call(), callcode(), delegatecall() and send(). The use of low
level calls should be avoided whenever possible, as if they fail they will not lead to a total reversion of
the current execution but return a boolean value set to false, while the code will continue to run. When
return values are not handled properly it can lead to unexpected behavior.
Definition 4. Unchecked Low Level Calls “If an external call fails, but is not checked, the contract will
continue execution as if the call succeeded. This will likely result in buggy and potentially exploitable
behavior from the contract.” [17]
Gassless send is a vulnerabilty defined by Atzei et al. that occurs when using the function send to
transfer ether to a contract, because it will then be possible to incur in an out-of-gas exception. As it
also relates to the use of the low level call send() we include this vulnerability also in this section.
Consider the example in Listing 2.11. The execution of send may fail in which case the winner will
not get the money, but prizePaidOut will be set to true. The condition in line 1 is always false and the
real winner can never claim the prize.
Listing 2.11: Vulnerable - send() can fail and prizePaidOut will be sent to true regardless.
1 if (gameHasEnded && !(prizePaidOut)) {2 winner.send(1000);3 prizePaidOut = true;4 }
21
Checking the return value of send to see if the operation is successfully and throwing an exception so
all the state is rolled back otherwise mitigates this issue, as seen in Listing 2.12.
Listing 2.12: Safe - prizePaidOut will be sent to true if send() is true.
1 if (gameHasEnded && !(prizePaidOut)) {2 if (winner.send(1000)){3 prizePaidOut = True;4 }5 else {6 throw;7 }8 }
In conclusion to mitigate this type of vulnerability the developer should manually perform validation
when making external calls and is encouraged the use of address.transfer() instead of address.send().
King of Ether [18] is a real world example of the impact of such vulnerabilities.
2.4.2.5 Denial of Service
There are many variants of DoS and they can be deadly to smart contracts while other types of appli-
cations may be able to recover. This type of attack consists in overwhelming the network with time-
consuming computations where users can leave the contract inoperable for a small period of time or
permanently.
Definition 5. Denial of Service “A conditional statement (if , for , while) should not depend on an
external call: the callee may permanently fail (throw or revert), preventing the caller from completing the
execution.” [6]
A Denial of Service attack can have multiple scenarios. Imagine an auction contract where bid(), seen
in Listing 2.13, refunds the frontrunner if he is outbid. When it tries to refund the old leader (line 3), it
reverts if the refund fails. This would allow a malicious bidder to become the leader while making sure
that any refunds to his address will always fail. The auction is then stalled which allows the attacker to
become the winner.
Listing 2.13: Vulnerable - send may fail and no one else will be able to bid.
1 function bid() public payable {2 require(msg.value > highestBid);
3 require(currentLeader.send(highestBid));
4 currentLeader = msg.sender;5 highestBid = msg.value;6 }
22
A recommendation is to set up a pull payment system instead, as shown in Listing 2.14. In this
example refunds are stored in a mapping to fight DoS, avoiding pushing balance to users favoring a pull
architecture. Execution fees can also protect against DoS attacks.
Listing 2.14: Safe - the refund is recorded safely.
1 function bid() payable external {2 require(msg.value >= highestBid);3 if (currentLeader != address(0)) {4 refunds[currentLeader] += highestBid;5 }6 currentLeader = msg.sender;7 highestBid = msg.value;8 }9
10 function withdrawRefund() external {11 uint refund = refunds[msg.sender];12 refunds[msg.sender] = 0;13 msg.sender.send(refund);14 }
As explained in [19]: “External calls can fail accidentally or deliberately. To minimize the damage
caused by such failures, it is often better to isolate each external call into its own transaction that can be
initiated by the recipient of the call.”. Pull over push should be favored for external calls and multiple
send() calls should be avoided in a single transaction. GovernMental [16], and old Ponzi scheme, was
susceptible to a DoS attack.
2.4.2.6 Bad Randomness
Also known as Nothing is Secret or Entropy Illusion. Pseudorandom number generation (PRNG) on the
blockchain is generally unsafe. The blockchain does not provide any cryptographically secure source of
randomness, everything in a contract is publicly visible and computers will always be faster than the
blockchain. While there are a variety of functions and variables that can access at first sight hard-to-
predict values, usually they are not that straightforward and can be more public than they seem or can
be subject to miners influence.
Definition 6. Bad Randomness “All transactions on the Ethereum blockchain are deterministic state
transition operations. This ultimately means that inside the blockchain ecosystem there is no source of
entropy or randomness.” [16]
In Listing 2.15 a private seed is used in combination with an iteration number and a keccak256 hash
function to determine if the caller wins. Even though the seed is private, it must have been set via a
transaction at some point in time and thus is visible on the blockchain.
23
Listing 2.15: Vulnerable to bad randomness.
1 uint256 private seed;2
3 function play() public payable {4 require(msg.value >= 1 ether);5 iteration++;
6 uint randomNumber = uint(keccak256(seed + iteration));7 if (randomNumber % 2 == 0) {8 msg.sender.transfer(this.balance);9 }
10 }
Currently there are not any straight forward recommended mitigations for this issue, although the
problem may be solved using on timed commitment protocols or external oracles [16]. SmartBillions [20],
a fully decentralized lottery system, got hacked exploiting this vulnerability.
2.4.2.7 Front Running
Also known as time-of-check vs time-of-use (TOCTOU), race condition or transaction ordering depen-
dence (TOD). Front running problems arise when two dependent transactions that invoke the same
contract are included in one block. The order of executing transactions relies on the miner. A malicious
user can launch an attack if those transitions were not executed in the right order. Unpredictable
state, defined by Atzei et al., also suits this category.
Definition 7. Front Running “There is a gap between the creation of a transaction and the moment it
is accepted in the blockchain. Therefore, an attacker can take advantage of this gap to put a contract in
a state that advantages them.” [17]
Transactions are only considered valid once a miner validates them. The miner who solves the block
also chooses which transactions from the pool will be included in the block. Imagine the contract in
Listing 2.16 contains 1000 Ether. The user who submits the solution, the pre-image of the hash (some
sha3 hash), wins the 1000 Ether. Lets say that a user discovers the solution and passes it as a parameter
in solve(). If an attacker is watching the transaction pool for anyone submitting a solution he is able
to see the solution and submit an equivalent transaction with a much higher gasPrice than the original
transaction. The miner who solves the block will likely give preference to the attacker transaction due to
its higher gasPrice and accept this transaction before the original one, allowing the attacker to win the
1000 Ether.
24
Listing 2.16: Vulnerable to Front Running.
1 contract FindThisHash {2
3 bytes32 constant public hash = somesha3hash;4 constructor() public payable {}5
6 function solve(string solution) public {7 require(hash == sha3(solution));8 msg.sender.transfer(1000 ether);9 }
10 }
Protecting against this is difficult, as it would come down to the specific contract itself. A robust
method is to use a commit-reveal scheme, whenever possible. ERC20 [21] and Bancor [22] are two real
world examples.
2.4.2.8 Time manipulation
Also referred as Timestamp Dependence or Time Constraints. The timestamp of the block can be
manipulated by the miner, and all direct and indirect uses of the timestamp should be considered.
Definition 8. Time Manipulation “Contracts can retrieve the timestamp in which the block was mined;
all the transactions within a block share the same timestamp. This guarantees the coherence with the
state of the contract after the execution, but it may also expose a contract to attacks, since the miner who
creates the new block can choose the timestamp with a certain degree of arbitrariness.” [1]
The function shown is Listing 2.17 only accepts calls that come after a specific date. As described
above, miners are able to influence their block’s timestamp and so attempt to mine a block containing
their transaction with a block timestamp set in the future. If accepted on the network the transaction will
give the miner ether before any other player could have attempted to win the game. Each block is sealed
by a miner at a precise time, and the current time can be displayed in Solidity with block.timestamp or
now.
Listing 2.17: Vulnerable to time manipulation.
1 function play() public {2 require(now > 1521763200 && neverPlayed == true);3 neverPlayed = false;4 msg.sender.transfer(1500 ether);5 }
The developer should be aware that the timestamp of the block can be manipulated by a miner and
should not be used for entropy or generating random numbers. The 15-second Rule - “If the contract
25
function can tolerate a 15-second drift in time, it is safe to use block.timestamp” [19] should be considered
and the use of block.number as a timestamp avoided.
GovernMental [16], referred before as susceptible to DoS attacks, is also vulnerable to a timestamp-
based attack.
2.4.2.9 Short Addresses
Short address attacks are possible due to the EVM itself accepting incorrectly padded arguments. It can
be exploited using specially-crafted addresses to make poorly coded clients encode arguments incorrectly
before including them in transactions.
Definition 9. Short Addresses “When passing parameters to a smart contract, the parameters are encoded
according to the ABI specification. It is possible to send encoded parameters that are shorter than the
expected parameter length.” [16]
Validating all inputs before sending them to the blockchain is thus crucial and will prevent these kinds
of attacks.
The GNT transfer bug [23], was a real world impact example of these type of vulnerabilities.
2.4.2.10 Unknown Unknowns
Smart contracts development is still in embryonic phase, its implementation and use are still very recent.
As well as the platforms themselves and the programming languages used to write smart contracts. New
classes of vulnerabilities will continue to be found and developers should be informed and up to date.
Best practices and documentation will need to be developed to ensure a better implementation.
26
2.5 Static Analysis Tools
In this section we present an overview of the state of the art in static analysis tools aimed at discovering
security issues in Ethereum smart contracts. First we briefly overview some of the methods used to
perform static analysis, then we overview of the state-of-the-art in smart contracts analysis tools. Later
we present a inclusion criteria that characterize the tools that we investigate further.
2.5.1 Methods
Static analysis refers to a set of techniques used to examine source code or bytecode without executing
it. The ultimate goal of static analysis in smart contracts is to automatically inspect the source code of
the contracts and find possible security issues. Static analysis techniques are potentially able to cover all
the execution paths of a smart contract and additionally can prove formally the absence or presence of a
bug. We briefly overview some of the techniques used in static analysis below.
Abstract Interpretation is a static analysis method used to prove behavioural properties of a
program. Certain instructions or certain effects of instructions are ignored while the bytecode is executing,
abstracting them away. Usually this process is done by translating instructions to another formalism and
then exploring all possible execution paths. Abstract interpretation was formalized by Patrick Cousot
as: “A program denotes computations in some universe of objects. Abstract interpretation of programs
consists in using that denotation to describe computations in another universe of abstract objects, so
that the results of abstract execution give some informations on the actual computations.” [24].
Constraint Solving aims to determine the solvability of constraints and tries to compute a concrete
solution. The goal is to find a solution to a set of constraints that impose conditions that the variables
must satisfy. Static analysis based on constraint solving consists in transforming a program and the
properties of interest into a set of constraints that are subsequently solved by a constraint solver.
Lexical Analysis groups source code syntax into sets of units that represent a meaningful syntax,
usually referred as ‘tokens’. It attempts to abstract the source code and make it easier to manipulate.
Static analysis performed using lexical analysis usually consists in the detection of certain syntactic
patterns.
Model Checking is a technique to verify the correctness of properties of finite-state systems. It
consists in checking exhaustively and automatically if the model meets a given specification, and thus if
correctness is verified.
Symbolic Execution refers to the execution of code using symbols instead of concrete values for
the variables. The goal is to determine what inputs cause each part of a program to execute. Symbolic
execution executes a program abstractly and can simultaneously explore multiple paths that a program
could take under different inputs. It is often the case that tools mix symbolic execution with other
techniques, such as abstract interpretation.
27
2.5.2 State Of The Art
The list of smart contract automated analysers presented below was collected in the context of the study
presented in the Chapter 4, where the selected tools in this Chapter are reviewed. We enumerate the
tools available and briefly discuss some of the tools that meet the criteria presented.
The survey of Angelo et al. [25] was used as a research starting point and we extended their list of
tools by searching academic literature and the internet for other tools. Through our research we were
able to find more eight tools that suited our discussion. We ended up with the 35 static analysis tools
that are listed below in Table 2.4.
# Tools Tool URLs1 contractLarva [26] https://github.com/gordonpace/contractLarva
2 E-EVM [27] https://github.com/pisocrob/E-EVM
3 Echidna https://github.com/crytic/echidna
4 Erays [28] https://github.com/teamnsrg/erays
5 Ether [29] N/A6 Ethersplay https://github.com/crytic/ethersplay
7 EtherTrust [30] https://www.netidee.at/ethertrust
8 EthIR [31] https://github.com/costa-group/EthIR
9 FSolidM [32] https://github.com/anmavrid/smart-contracts
10 Gasper [33] N/A11 HoneyBadger [34] https://github.com/christoftorres/HoneyBadger
12 KEVM [35] https://github.com/kframework/evm-semantics
13 MadMax [36] https://github.com/nevillegrech/MadMax
14 Maian [4] https://github.com/MAIAN-tool/MAIAN
15 Manticore [37] https://github.com/trailofbits/manticore/
16 Mythril [38] https://github.com/ConsenSys/mythril-classic
17 Octopus https://github.com/quoscient/octopus
18 Osiris [39] https://github.com/christoftorres/Osiris
19 Oyente [5] https://github.com/melonproject/oyente
20 Porosity [40] https://github.com/comaeio/porosity
21 rattle https://github.com/crytic/rattle
22 ReGuard [41] N/A23 Remix https://github.com/ethereum/remix
24 SASC [42] N/A25 sCompile [43] N/A26 Securify [44] https://github.com/eth-sri/securify
27 Slither [45] https://github.com/crytic/slither
28 Smartcheck [6] https://github.com/smartdec/smartcheck
29 Solgraph https://github.com/raineorshine/solgraph
30 Solhint https://github.com/protofire/solhint
31 SolMet [46] https://github.com/chicxurug/SolMet-Solidity-parser
32 teEther [47] https://github.com/nescio007/teether
33 Vandal [48] https://github.com/usyd-blockchain/vandal
34 VeriSol [49] https://github.com/microsoft/verisol
35 Zeus [50] N/A
Table 2.4: Tools identified and their URLs
28
Not all the identified tools are well suited for our discussion. Only the tools that met the following
five inclusion criteria are relevant to be discussed:
• Criterion #1. [Available and CLI] The tool is publicly available and supports a command-line interface
(CLI). The CLI facilitates the scalability of the analyses, discussed in Chapter 4
• Criterion #2. [Compatible Input] The tool takes as input a Solidity contract. This excludes tools that
only consider EVM bytecode.
• Criterion #3. [Only Source] The tool requires only the source code of the contract to be able to run
the analysis. This excludes tools that require a test suite or contracts annotated with assertions.
• Criterion #4. [Vulnerability Finding] The tool identifies vulnerabilities or bad practices in contracts.
This excludes tools that are described as analysis tools, but only construct artifacts such as control
flow graphs.
• Criterion #5. [Static Analysis] The tool only executes static analysis techniques to find security issues.
After inspecting all 35 analysis tools presented in Table 2.4, Table 2.6 defines the 7 tools that meet the
inclusion criteria outlined. Table 2.5 presents the excluded tools and which criteria they did not meet.
Inclusion criteria Tools that violate criteriaAvailable and CLI (C1) Ether, Gasper, ReGuard, Remix, SASC, sCompile, teEther,
ZeusCompatible Input (C2) MadMax, VandalOnly Source (C3) Echidna, VeriSolVulnerability Finding (C4) contractLarva, E-EVM, Erays, Ethersplay, EtherTrust, EthIR,
FSolidM, KEVM, Octopus, Porosity, rattle, Solgraph, SolMet,Solhint
Static Analysis (C5) contractLarva, Echidna, MAIAN, Manticore, ReGuard
Table 2.5: Excluded analysis tools based on our inclusion criteria.
Tools that meet criteria HoneyBadger, Mythril, Osiris, Oyente, Securify, Slither,Smartcheck
Table 2.6: Included analysis tools based on our inclusion criteria.
HoneyBadger [34] is an Oyente-based (see below) tool that employs symbolic execution and a set
of heuristics to pinpoint honeypots in smart contracts. Honeypots are smart contracts that appear to
have an obvious flaw in their design, which allows an arbitrary user to drain Ether from the contract,
given that the user transfers a priori a certain amount of Ether to the contract. When HoneyBadger
29
detects that a contract appears to be vulnerable, it means that the developer of the contract wanted to
make the contract look vulnerable, but is not vulnerable.
Mythril [38] analyses EVM bytecode and relies on concolic analysis, which is a hybrid analysis
technique that performs symbolic execution along a concrete execution path, to find several types of
security issues. Like Oyente, it uses Z3 Theorem prover in conjunction with the symbolic execution
engine laser-ethereum.
Osiris [39] extends Oyente to improve the detection of integer bugs. It combines symbolic execution
and taint analysis, in order to accurately find integer bugs in Ethereum smart contract.
Oyente [5] is one of the first static analysis tool for Ethereum smart contracts and has been forked by
several other projects, including HoneyBadger and Osiris. It runs symbolic execution on EVM bytecode
to identify vulnerabilities. Determines which inputs cause which program branches to execute to find
potential security vulnerabilities and it uses Z3 Theorem prover to decide satisfiability of the explored
paths. Oyente works directly with EVM bytecode without access high level representation.
Securify [44] statically analyses EVM bytecode to infer relevant and precise semantic information
about the contract using the Souffle Datalog solver. It then checks compliance and violation patterns
that capture sufficient conditions for proving if a property holds or not.
Slither [45] is a static analysis framework that converts Solidity smart contracts into an intermediate
representation called SlithIR and applies known program analysis techniques such as dataflow and taint
tracking to extract and refine information.
Smartcheck [6] is a static analysis tool that looks for vulnerability patterns and bad coding practices.
It runs lexical and syntactical analysis on Solidity source code.
30
3SmartBugs: An Execution
Framework
Contents
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4 Framework Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5 Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
31
32
In this chapter we introduce SmartBugs, a new extendable execution framework created to facilitate
the integration and comparison between multiple analysis tools and the analysis of Ethereum smart
contracts. We also present sbcurated, a curated dataset of vulnerable smart contracts written in Solidity,
easily reproducible and constructed to facilitate comparison of research that comes built in SmartBugs.
We briefly discuss our objectives and the methodologies used to choose contracts, filter bugs, and add
tools.
3.1 Motivation
There has been some effort from the research community to develop automated analysis tools that locate
and eliminate vulnerabilities in smart contracts [5, 6, 44, 51]. However, it is not easy to compare and
reproduce that research: even though several of the tools are publicly available, the datasets used are
not. If a developer of a new tool wants to compare the new tool with existing work, the current approach
is to contact the authors of alternative tools and hope that they give access to their datasets (as done in,
e.g., [52]). Furthermore, most of smart contracts static analysers are required to be installed, and some
even require you to install dependencies they rely on. Others are not available for all OS’s. Executing
smart contract analysis with multiple tools, running and installing each one individually can be an
excruciating process and consume unnecessary time. There is a need to ease the execution of static
analysers and provide a simple interface where the user could analyse multiple contracts with multiple
tools without requiring installation of any of the tools.
We present SmartBugs to address these problems. SmartBugs is an extensible, and easy to use
execution framework that simplifies research on automated analysis techniques for smart contracts and
executes these tools on the same execution environment. To be able to execute and compare automated
analysis tools, hence setting the ground for fair comparisons, we provide sbcurated: a dataset of 143
manually annotated Solidity smart contracts that can be used to evaluate the precision of analysis tools.
Developers can use it to evaluate their automated analysis tools, create benchmarks and easily compare
them with state of the art, alternative tools. SmartBugs provides the possibility to easily integrate analysis
tools, thus enabling reproducibility of the results for these tools. Once tools are added to SmartBugs, it
is easy to compare their performance and accuracy.
SmartBugs has the following features:
• A plugin system to easily add new analysis tools, based on Docker images;
• A plugin system to easily add new named datasets;
• Bulk analysis with any number of tools available;
• CLI and WUI Dashboard;
33
• Parallel execution of the tools to speed up the execution time; [Contribution]
• A parser mechanism that normalizes the way the tools are outputting the results; [Contribution]
The last two items marked with [Contribution] resulted in external contributions done to SmartBugs
by co-authors of Durieux et al. [7] to automate results in the ambit of the analysis discussed in Chapter
4.
SmartBugs comes with 10 tools ready to be executed:
1. HoneyBadger
2. Maian
3. Manticore
4. Mythril
5. Osiris
6. Oyente
7. Securify
8. Slither
9. SmartCheck
10. Solhint
However, throughout this work we only focus on the seven tools that met the criteria presented in
Section 2.5. We excluded Maian, Manticore and Solhint.
3.2 Availability
The framework and sbcurated are available in GitHub as repositories of SmartBugs 12. The dataset
sbcurated is divided into 10 folders named with the DASP categories, and the folders contain the contracts
of that category. Moreover, the dataset contains the file vulnerabilities.json which contains the
details of each vulnerable contract. It details the name, the origin URL, the path, and the lines and the
category of the vulnerabilities.
3.3 Data Collection
All contracts and tools present in SmartBugs are publicly available. The collection methodology to create
sbcurated is explained in this section.
1SmartBugs (including sbcurated): https://github.com/smartbugs/smartbugs2SmartBugs WUI: https://github.com/smartbugs/smartbugs-dashboard
34
3.3.1 sbcurated: A Dataset of 143 Vulnerable Smart Contracts
As referenced in the beginning of this chapter, our goal in constructing sbcurated was to provide a
reliable dataset with a collection of vulnerabilities designed to be reproducible, follow a known taxonomy
(as presented in Section 2.4) and to serve as a reference dataset to the research community. The last
category presented in DASP 10 is Unknown Unknowns, which, as discussed in Section 2.4 represents
future and undiscovered vulnerabilities. This category is not useful in the context of mapping existent
vulnerabilities, so we opted to map vulnerabilities that did not fit any other of the nine categories, i.e.
categories that are not identified in DASP 10, in this category. For example, vulnerabilities such as
uninitialized data and the possibility of locking down Ether are mapped to this category. This category,
Unknown Unknowns, will be referred as Other to avoid confusion and simplify the description.
sbcurated design was driven by the following objectives:
1. Reproducibility: One of our main objectives is to contribute to the reproducibility of research in
automated analysis of smart contracts and, in particular, of smart contracts written in Solidity. By
collecting, organizing, and maintaining a diverse set of vulnerable contracts, we already contribute
towards that objective.
2. Real-world relevance: The contracts included in sbcurated were all taken from public sources.
Some were specifically crafted to demonstrate certain vulnerabilities, but a considerable number
were taken from Etherscan (i.e., they were deployed).
3. Diversity: The contracts in sbcurated were collected from multiple public sources aimed at demon-
strating different types of vulnerabilities. It includes multiple representatives from the categories
listed by the DASP.
4. Custom filtering: Named sets are useful for debugging and for specific experiments that a devel-
oper or researcher may want to run. It can ease and automate the execution process and facilitate
benchmarks results.
sbcurated was created by collecting smart contracts from three different sources: GitHub repositories, Blog
posts that analyse contracts and the Ethereum network. Most of contracts were collected from GitHub
repositories and the Ethereum network. We identified GitHub repositories that match relevant search
queries (‘vulnerable smart contracts’, ‘smart contracts security’, ‘solidity vulnerabilities’ ) and contain
vulnerable smart contracts. We searched Google using the same queries. We found several repositories
with vulnerable smart contracts such as not-so-smart-contracts3 and SWC Registry4. The latter is a
classification scheme for security weaknesses in Ethereum smart contracts. We also extracted vulnerabil-
ities that come from trusted entities in blog posts where smart contracts are audited, tested or discussed
3not-so-smart-contracts: https://github.com/crytic/not-so-smart-contracts4SWC Registry: https://smartcontractsecurity.github.io/SWC-registry
35
such as Positive.com5 and Blockchain.unica6. And finally, we used Etherscan7 to collect smart contracts
that are deployed on the Ethereum network and are known to contain vulnerabilities (e.g. the original
SmartBillions contract). Note that all the contracts were collected from trusted entities in the field. We
also ensure the traceability of each contract by providing the URL from which they were taken and its
author, where possible.
3.3.1.1 sbcurated Statistics
The dataset contains 143 contracts and 191 manually tagged vulnerabilities, divided into 10 categories of
vulnerabilities. Table 3.1 presents information about the 143 contracts. Each line contains a category of
vulnerability. For each category, we provide the number of contracts available within that category and
the total number of vulnerabilities and number of lines of code in the contracts of that category.
Category Contracts Vulns LoC
Access Control 17 19 899Arithmetic 14 24 304Bad Randomness 8 31 1,079Denial of service 6 7 177Front running 4 7 137Reentrancy 31 32 2,164Short addresses 1 1 18Time manipulation 5 7 100Unchecked low level calls 53 60 4055Other 3 3 115
Total 143 191 9,048
Table 3.1: Categories of vulnerabilities available in the dataset. (computed using cloc 1.82).
3.4 Framework Architecture
SmartBugs is written Python3 and relies on docker images to execute the tools. SmartBugs works with
docker images available on the Docker Hub or locally. In order to be able to add a tool to SmartBugs, a
docker image of the tool must be available.
The choice to use docker images was made to ease the addition of tools, allow the execution to
be reproducible and keep the same execution environment for all tools, allowing the user to execute
SmartBugs in every environment where python3 and docker is installed.
5Positive.com: https://blog.positive.com6Blockchain.unica: http://blockchain.unica.it/projects/ethereum-survey/7Etherscan: https://etherscan.io
36
Figure 3.1: SmartBugs Architecture
SmartBugs is composed of five main parts plus a web-based user interface, that is integrated to interact
with SmartBugs, as seen in Figure 3.1. SmartBugs can then be separated and divided in the following
components:
• The command-line interface;
• The web-based user interface;
• The tool plugins;
• The Docker images that are stored on Docker Hub;
• The datasets of smart contracts, where sbcurated is included;
• The SmartBugs’ runner. The component that puts all the parts of SmartBugs together to execute
the analysis tools on the smart contracts.
3.4.1 System Requirements
SmartBugs requires Docker and Python3 with the modules PyYAML, solidity parser, and docker.
Since Solidity versions are not always retro-compatible, the tools currently available might have prob-
lems processing some contracts depending on the solidity compiler used. For example, Solidity v0.5.0
introduced breaking changes8 and this creates compatibility issues with some versions of the Mythril tool.
Indeed, Mythril and Securify have compatibility issues with specific Solidity versions since they analyse
directly the EVM bytecode. However, tools such as SmartCheck that analyse the source code are less
impacted by the changes between versions.
To mitigate this problem, SmartBugs provides the possibility of having two different versions of the
same tool by adding a property in the configuration file. The configuration file supports a default tool
version to compile and analyse contracts above Solidity v0.5.0 (or all contracts if no other tool version is
8Solidity v0.5.0 introduced breaking changes: https://solidity.readthedocs.io/en/v0.5.0/050-breaking-changes.
html
37
provided). It is also possible to specify a different tool version to compile and analyse contracts below
Solidity v0.5.0. For example, tools such as Mythril and Securify, have two distinct versions configured:
one for contracts with Solidity below v0.5.0 and other for contracts written with Solidity above v0.5.0.
The design decision that we took was to put the users in control, allowing them to adapt the dataset
to their own needs.
3.4.2 Methodology for adding tools
Addition of tools in SmartBugs is designed to be simple and practical, allowing the user to control the
execution of the tools according to their needs. Currently, all the 10 tools in SmartBugs use Docker
images pulled from the Docker Hub. Some images were already publicly available others were created to
be added to SmartBugs.
Each tool plugin contains the configuration of the tools. The configuration contains the name of the
Docker image, the name of the tool, the command line to run the tool, and, optionally, the description
of the tool and the location of the output of results. Once a docker image providing the tool is available,
adding the tool to SmartBugs consists of adding a new configuration (an .YAML file) file such as the
following:
docker_image:
default: qspprotocol/mythril-usolc
solc<5: qspprotocol/mythril-0.4.25
cmd: -x
info: Mythril analyses EVM bytecode using symbolic analysis, taint analysis and control flow \
checking to detect a variety of security vulnerabilities.
SmartBugs extracts the results from the output printed by tool by default. If instead a tool stores
the result of the analysis in a file on the docker images, the path of that file should be defined in the
configuration file by adding an extra field, as shown below:
docker_image:
default: qspprotocol/securify-usolc
solc<5: qspprotocol/securify-0.4.25
cmd: --livestatusfile /results/output.json -fs
output_in_files:
folder: /results/output.json
info: Securify uses formal verification, also relying on static analysis checks. Securify’s \
analysis consists of two steps. First, it symbolically analyses the contract’s dependency \
38
graph to extract precise semantic information from the code. Then, it checks compliance and \
violation patterns that capture sufficient conditions for proving if a property holds or not.
3.4.3 Methodology for filtering bugs
SmartBugs supports the definition of named datasets, which is intended to represent subsets of contracts
that share a common property. For example, a named dataset already provided by default is reentrancy : it
corresponds to contracts that are identified as being vulnerable to reentrancy attacks. Named datasets can
be specified in a configuration file (config/dataset/dataset.yaml). The default configuration file provides
the named sets shown in Table 3.1.
To add a custom named dataset, the user simply has to alter the configuration file by adding a name
and the correspondent list of paths. The path can be a directory, a file, or a list of both. An example is
shown below.
reentrancy: dataset/reentrancy
arithmetic:
- dataset/arithmetic
- dataset/reentrancy/reentrance.sol
The dataset configuration file defines the name of the datasets and the path to the correspondent
directories or files.
3.4.4 CLI
SmartBugs provides a command line interface that allows users to query the datasets defined and run
different analysis tools on sets of contracts. The user can also get information about the tools, if provided,
skip an execution that already has results, specify the number of processes to use during the analysis (by
default 1) and list the named datasets and tools available.
SmartBugs command-line interface can be invoked as:
smartBugs.py [-h, --help]
(--file FILES | --dataset DATASETS)
--tool TOOLS
--info TOOLS
--skip-existing
--processes PROCESSES
--list tools datasets
39
Usage Example The following example shows how we can use the CLI of SmartBugs to run the tools
Oyente and Mythril against the contracts in the named dateset reentrancy :
smartBugs.py --tool oyente mythril \
--dataset reentrancy
This command creates an output folder with the results of the analysis for each tool executed. By
inspecting the output files, we can determine very quickly that both tools found problems in exactly the
same three contracts, further increasing confidence that the analyses performed by these tools are correct.
Since all 10 tools added to SmartBugs come with a parser mechanism to normalize the output, a json
file, with all vulnerabilities detected by the tool is created. A file containing the raw output of the tool
executed is also generated in the same folder. Also, the SmartBugs logs are stored in a folder called logs
composed of files named after the date and hour of the execution.
3.4.5 WUI Dashboard
SmartBugs offers a Web-based UI (WUI) built on top of SmartBugs itself. This dashboard provides the
user easy access to the list of tools, named datasets available and the vulnerabilities detected by each
tool available mapped to a category of DASP 10.
SmartBugs WUI is not dynamically updated, meaning that if new tools or datasets are added to
SmartBugs, they will not be automatically available on the WUI (unlike CLI). The reason for this is that
there is a need to compute the outputs and thus a parser mechanism must be defined in order to present
results to the user. Instead, to add a tool or dataset, first the user should follow the procedure explained
in 3.4.2 and 3.4.3 and then manually update the WUI by adding the name of the tool or the dataset. It
should also be defined a way to parse the output. The dashboard currently supports 10 tools, includes
sbcurated and provides a detailed mapping of all vulnerabilities identified by the 10 tools to a DASP 10
category.
The user has three options to analyse a smart contract, as seen in Figure 3.2:
1. The user can paste or write a smart contract in a text box;
2. The user can import a smart contract by uploading it to SmartBugs;
3. The user can run tools on defined datasets;
After the execution of the analysis the dashboard will present a graph with the number of security
issues found by each tool, and for each tool present the issues found.
40
Figure 3.2: SmartBugs Dashboard
Usage Example The following figures show how we can use the WUI of SmartBugs to run the tools
Oyente, Slither and SmartCheck against the contract inputted and how the results are shown:
Figure 3.3: Input panel for smart contracts and tool’s choice (Oyente, Slither and SmartCheck are selected)
41
Figure 3.4: Number of issues found by each tool
Figure 3.5: Issues found by Oyente
For each tool executed, a similar output to Figure 3.5 is presented.
3.5 Use Case
SmartBugs was used to perform an empirical review of automated analysis tools on 47,587 Ethereum
smart contracts further discussed in Chapter 4.
42
4Analysis: An Empirical Review
Contents
4.1 Analysis of 47,587 Smart Contracts . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Precision of the Static Analysis Tools . . . . . . . . . . . . . . . . . . . . . . 53
43
44
In this Chapter we discuss and analyse the static analysis tools introduced in Section 2.5 using two
distinct datasets, sbcurated and sbwild.
The first section of this chapter (Section 4.1) is part of a paper submitted to ICSE 2020 1 [7]. This
paper is authored by the author of this Dissertation and three researchers from INESC-ID. The content
of the first section is credited to all for the authors. (Note: In the paper were also included two tools
that do not rely solely on static analysis, which are not taken into account in this Dissertation).
4.1 Analysis of 47,587 Smart Contracts
We used SmartBugs to execute the 7 automated analysis tools on two datasets. The first dataset is
sbcurated, introduced in the previous chapter. At the time of the execution of this study sbcurated was
composed of 69 Solidity smart contracts with 115 manually annotated vulnerabilities (as described in
Table 4.2), not 143 smart contracts as presented in Chapter 3. The first part of the study was then
be executed on 69 vulnerable smart contracts and throughout this chapter all sbcurated mentions are
referring to this version. The second dataset, coined sbwild, contains all available smart contracts in the
Ethereum Blockchain that have Solidity source code available on Etherscan2 (a total of 47,518 contracts).
We have executed 7 state-of-the-art automated static analysis tools on the two datasets and analysed
the results in order to provide a fair point of comparison for future smart contract analysis tools. In
total, the execution of all the tools required approximately 330 days and 15 hours to complete 333,109
analyses.
In this section, we describe the setup of the tools (Section 4.1.2), their execution (Section 4.1.3) and
discuss the results of the analysis of both datasets (Section 4.1.4 and 4.1.5).
4.1.1 Data Collection: sbwild
The dataset sbwild was collected by the co-authors of [7], and is further detailed in the paper. It collected
all the different contracts from the Ethereum blockchain, since Etherscan allows downloading the source
code of a contract, if available, if you know its address. All the duplicates were removed as we observed
that 95% of the available Solidity contracts are duplicates. We consider that two contracts are duplicates
when the MD5 checksums of the two source files are identical after removing all the spaces and tabulations.
1ICSE 2020: https://conf.researchr.org/home/icse-20202Etherscan: https://etherscan.io
45
Solidity source not available 1,290,074Solidity source available 972,975Unaccessible 47
Total 2,263,096
Unique Solidity Contracts 47,518LOC 9,693,457
Table 4.1: Statistics on the collection of Solidity smart contracts from the Ethereum blockchain.
Table 4.1 presents the statistics of this dataset. We use Google BigQuery 3 to collect the Ethereum
contract addresses that have at least one transaction. The query on Google BigQuery retrieved 2,263,096
smart contract addresses. We then requested Etherscan for the Solidity source code of those contracts,
and we obtained 972,975 Solidity files. This means that 1,290,074 of the contracts do not have an
associated source file in Etherscan. The filtering process of removing duplicate contracts resulted in
47,518 unique contracts (a total of 9,693,457 lines). The dataset sbwild is available in GitHub as a
repository of SmartBugs 4.
4.1.2 Tools’ Setup
For this experiment, we set the time budget to 30 minutes per analysis. In order to identify a suitable time
budget for one execution of one tool over one contract, we first executed all the tools on sbcurated dataset.
We then selected a time budget that is higher than the average execution time. If the time budget is
spent, we stop the execution and collect the partial results of the execution. During the execution of our
experiment, no tool faced timeouts.
4.1.3 Large-scale Execution
To our knowledge, the experimental setup presented in [7] is the largest one on smart contract analysis,
both in the number of tools and in execution time.
Regarding the study presented in this section, where two tools were excluded from the original study,
7 tools were executed and 333,109 analyses were performed, which took approximately 330 days and 15
hours of combined execution, almost a year of continuous execution.
All the logs and the raw results of the analysis are available in GitHub as a repository of SmartBugs 5.
4.1.4 sbcurated: Analysis of 69 Smart Contracts
Each contract of sbcurated is categorized in one of the ten DASP categories, where Unknown Unknowns,
referenced as Other in this section, represents vulnerabilities that did not suited any of the other nine
3Ethereum in BigQuery - a Public Dataset for smart contract analytics:https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-public-dataset-smart-contract-analytics
4Analysis results: https://github.com/smartbugs/smartbugs-wild5Analysis results: https://github.com/smartbugs/smartbugs-results
46
categories in the DASP 10, as established in the previous Chapter. The version of sbcurated used is
described in Table 4.2. Our goal is to compute the ability of the 7 tools in detecting the vulnerabilities
present in the 69 contracts. The methodology that we followed to extract results was the following:
1. We executed the 7 tools on the 69 contracts. The result of this execution is available on GitHub.
2. We extracted all the vulnerabilities that were detected by the tools into a JSON file.
3. We mapped the detected vulnerabilities by the tools to a category of vulnerabilities (see Table
3.1). To achieve this task, we used all the annotated vulnerability types that sbcurated provides.
For example, Oyente detects a vulnerability called Integer Overflow that we link to the category
Arithmetic. In total, we identify 141 vulnerability types, and 97 of them have been tagged in one
of the ten categories. The remaining 44 do not fit the DASP taxonomy (for example, some tools
warn about the use of inline assembly, which is considered a bad practice but does not necessarily
lead to vulnerable contracts). The mapping that we created is publicly available6 and can be also
consulted in SmartBugs Dashboard.
4. At this point, we were able to identify which vulnerabilities the tools detect. Unfortunately, we found
out that none of the 7 tools were able to detect vulnerabilities of the categories Bad Randomness
and Short Addresses.
Category Contracts Vulns LoC
Access Control 17 19 899Arithmetic 14 22 295Bad Randomness 8 31 1,079Denial of service 6 7 177Front running 4 7 137Reentrancy 7 8 778Short addresses 1 1 18Time manipulation 4 5 76Unchecked low level calls 5 12 225Other 3 3 115
Total 69 115 3,799
Table 4.2: Categories of vulnerabilities available in the dataset sbcurated used in the study. (computed usingcloc 1.82).
6Vulnerabilities mapping: https://tinyurl.com/smartbugs-mapping
47
Category HoneyBadger Mythril Osiris Oyente Securify Slither SmartCheck Total
Access Control 0/19 0% 4/19 21% 0/19 0% 0/19 0% 0/19 0% 4/19 21% 2/19 11% 5/19 26%Arithmetic 0/22 0% 15/22 68% 11/22 50% 12/22 55% 0/22 0% 0/22 0% 1/22 5% 19/22 86%Denial of Service 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/ 7 0%Front Running 0/7 0% 2/7 29% 0/7 0% 0/7 0% 2/7 29% 0/7 0% 0/7 0% 2/ 7 29%Reentrancy 0/8 0% 5/8 62% 5/8 62% 5/8 62% 5/8 62% 7/8 88% 5/8 62% 7/ 8 88%Time Manipulation 0/5 0% 0/5 0% 0/5 0% 0/5 0% 0/5 0% 2/5 40% 1/5 20% 3/ 5 60%Unchecked Low Level Calls 0/12 0% 5/12 42% 0/12 0% 0/12 0% 3/12 25% 4/12 33% 4/12 33% 9/12 75%Other 2/3 67% 0/3 0% 0/3 0% 0/3 0% 0/3 0% 3/3 100% 0/3 0% 3/ 3 100%
Total 2/115 2% 31/115 27% 16/115 14% 17/115 15% 10/115 9% 20/115 17% 13/115 11% 48/115 42%
Table 4.3: Vulnerabilities identified per category by each tool.
Category HoneyBadger Mythril Osiris Oyente Securify Slither SmartCheck Total
Access Control 0 24 0 0 6 20 3 53Arithmetic 0 92 62 69 0 0 23 246Denial of Service 0 0 27 11 0 2 19 59Front Running 0 21 0 0 55 0 0 76Reentrancy 0 16 5 5 32 15 7 80Time Manipulation 0 0 4 5 0 5 2 16Unchecked Low Level Calls 0 30 0 0 21 13 14 78Other 5 32 0 0 0 28 8 73
Total 5 215 98 90 114 83 76 681
Table 4.4: Total number of detected vulnerabilities by each tool, including vulnerabilities not tagged in thedataset.
The results of the analysis of sbcurated are presented in Table 4.3 and Table 4.4. The first table
presents the number of known vulnerabilities that have been identified. A vulnerability is considered as
identified when a tool detects a vulnerability of a specific category at a specific line, and it matches the
vulnerability that has been annotated in the dataset. Each row of Table 4.3 represents a vulnerability
category, and each cell presents the number of vulnerabilities where the tool detects a vulnerability of this
category. This table summarizes the strengths and weaknesses of the current state of the art of smart
contract analysis tools. It shows that the tools can accurately detect vulnerabilities of the categories
Arithmetic, Reentrancy, Time Manipulation, Unchecked Low Level Calls and Other. However, they were
not accurate in detecting vulnerabilities of the categories Access Control, Denial of Service, and Front
Running. The categories Bad Randomness and Short Addresses are not listed, since none of the tools
are able to detect vulnerabilities of these types. This shows that there is still room for improvement and,
potentially, for new approaches to detect vulnerabilities of the ten DASP categories.
Table 4.4 also shows that the tools offer distinct accuracies. Indeed, the tool Mythril has the best
accuracy among the 7 tools. Mythril detects 27% of all the vulnerabilities when the average of all tools
is 12%. The ranking of the tools is comparable to the one observed by Parizi et al. [53]. However, the
average accuracy is lower on our benchmark sbcurated. Moreover, Mythril, Slither, and SmartCheck are
the tools that detect the largest number of different categories (5 categories). Despite its good results,
Mythril is not powerful enough to replace all the tools: by combining the detection abilities of all the
tools, we succeed to detect 42% of all the vulnerabilities. However, depending on the available computing
power, it might not be realistic to combine all the tools.
48
HoneyBadger Mythril Osiris Oyente Securify Slither SmartCheck
HoneyBadger 33/115 29% 18/115 16% 19/115 17% 12/115 10% 20/115 17% 15/115 13%Mythril 33/115 29% 33/115 29% 31/115 27% 42/115 37% 33/115 29%Osiris 22/115 19% 21/115 18% 31/115 27% 23/115 20%Oyente 22/115 19% 32/115 28% 25/115 22%Securify 25/115 22% 16/115 14%Slither 25/115 22%SmartCheck
Table 4.5: Combined tools
Therefore, we suggest the combination of Mythril and Slither, since it detects 42 (37%) of all the
vulnerabilities (see Table 4.5). This combination offers a good balance between performance and execution
cost. This combination is the best possible combination by a considerable margin. The second best
combination, Mythrill and Oyente, only succeeds to detect 33 (29%) of all the vulnerabilities.
We now consider all the vulnerability detections and not only the ones that have been tagged in
sbcurated. Table 4.4 presents the total number of vulnerabilities detected by the tools. This table al-
lows the comparison of the total number of detected vulnerabilities with the number of detected known
vulnerabilities shown in Table 4.3. Unsurprisingly, the more accurate a tool is in detecting known vulner-
abilities, the more accurate it is at detecting unknown vulnerabilities. The output of the tools that are
performing the best are also producing much more warnings (i.e., their output is more noisy), making it
difficult for a developer to exploit the results.
4.1.5 sbwild: Vulnerabilities in Production Smart Contracts
The analysis of the contracts in sbwild followed the same methodology as in the previous analysis of
sbcurated, however, for sbwild, we do not have an oracle to identify the vulnerabilities.
Category HoneyBadger Mythril Osiris Oyente Securify Slither SmartCheck Total
Access Control 0 0% 1,076 2% 0 0% 2 0% 614 1% 2,356 4% 384 0% 3,758 7%Arithmetic 1 0% 18,515 39% 13,922 29% 34,306 72% 0 0% 0 0% 7,430 15% 37,590 79%Denial of Service 0 0% 0 0% 485 1% 880 1% 0 0% 2,555 5% 11,621 24% 12,419 26%Front Running 0 0% 2,015 4% 0 0% 0 0% 7,217 15% 0 0% 0 0% 8,161 17%Reentrancy 19 0% 8,454 17% 496 1% 308 0% 2,033 4% 8,764 18% 847 1% 14,747 31%Time Manipulation 0 0% 0 0% 1,470 3% 1,452 3% 0 0% 1,988 4% 68 0% 4,005 8%Unchecked Low Calls 0 0% 443 0% 0 0% 0 0% 592 1% 12,199 25% 2,867 6% 14,655 30%Other 26 0% 11,126 23% 0 0% 0 0% 561 1% 9,133 19% 14,113 29% 28,091 59%
Total 46 0% 22,994 48% 14,665 30% 34,764 73% 8,781 18% 22,269 46% 24,906 52% 44,549 93%
Table 4.6: The total number of contracts that have at least one vulnerability (analysis of 47,518 contracts).
Table 4.6 presents the results of executing the 7 tools on the 47,518 contracts. It shows that the 7
tools are able to detect eight different categories of vulnerabilities. In total, 44.549 contracts (93%) have
at least one vulnerability detected by one of the 7 tools.
Such a high number of vulnerable contracts suggests the presence of a considerable number of false
positives. Oyente is the approach that identifies the highest number of contracts as vulnerable (73%),
mostly due to vulnerabilities in the Arithmetic category. This observation is coherent with the observation
49
of Parizi et al. [53], since they determine that Oyente has the highest number of false positives when
compared to Mythril, Securify, and SmartCheck.
Figure 4.1: Proportion of vulnerabilities identified by exactly one (1), two (2) or three (3) tools, and by fourtools or more (4+)
Since we observed a potentially large number of false positives, we analysed to what extent the tools
agree in vulnerabilities they flag. The hypothesis is that if a vulnerability is identified exclusively by a
single tool, the probability of it being a false positive increases. Figure 4.1 presents the results of this
analysis. This figure shows the proportion of detected vulnerabilities that have been identified exclusively
by one tool alone, two tools, three tools, and finally by four or more tools.
It is clear from the figure that the large majority of the vulnerabilities have been detected by one tool
only. One can observe that there are 71.25% of the Arithmetic vulnerabilities to be found by more than
one tool.
We also observe that only a few number of vulnerabilities received a consensus of four or more tools:
937 for Arithmetic and 133 for Reentrancy. Unchecked Low Level Calls has consensus by four tools
or more in 52 vulnerabilities, Denial Of Service and Other in 50 and 41, respectively. These results
suggest that combining several of these tools may yield more accurate results, with less false positives
and negatives.
The tool HoneyBadger is different: instead of detecting vulnerabilities, it detects malicious contracts
that try to imitate vulnerable contracts in order to attract transactions to their honeypots. HoneyBad-
ger has a peculiar, but useful role: if HoneyBadger detects a vulnerability, it actually means that the
vulnerability does not exist. So, consensus with HoneyBadger suggests the presence of false positives.
Figura 4.1 shows that 15 contracts identified by HoneyBadger with vulnerabilities of type Reentrancy
have been detected by three other tools as Reentrancy vulnerable.
We also analysed the evolution of the vulnerabilities over time. Figure 4.2 presents the evolution of
the number of vulnerabilities by category. It firstly shows the total number of unique contracts started
50
Figure 4.2: Evolution of number of vulnerabilities over time.
to increase exponentially at the end of 2017 when Ether was at its highest value. Secondly, we can
observe two main groups of curves. The first one contains the categories Arithmetic and Other. These
two categories follow the curve of the total number of contracts. The growing number of vulnerable
contracts seems to slow down from July 2018. Finally, this figure shows that the evolution of categories
Reentrancy and Unchecked Low Level Calls is extremely similar (the green line of Reentrancy is also
hidden by the blue line of Unchecked Low Level Calls). This indicates that there is a strong correlation
between vulnerabilities in these two categories.
Figure 4.3: Correlation between the number of vulnerabilities and balance in Wei (one Ether is 1018 Wei)
And lastly, Figure 4.3 presents the correlation between the number of vulnerabilities and the balance
of the contracts. It shows that the contracts that have a balance between 1014 Wei and 1020 Wei have
more vulnerabilities than other contracts. Hence, the richest and the middle class seem to be most
51
impacted. Per category, we have not observed any significant differences worth reporting.
4.1.6 Execution Time of the Analysis Tools
In order to measure the time of the execution, we recorded for each individual analysis when it started
and when it ended. The duration of the analysis is the difference between the starting time and the
ending time. An individual execution is composed of the following steps:
1. Start the Docker image and bind the contract to the Docker instance;
2. Clean the Docker container;
3. Parse and output the logs to the results folder.
Table 4.7 presents the average and total times used by each tool. In the table, we can observe two
different groups of execution time: the tools that take a few seconds to execute and the tools that take
a few minutes. Oyente, Osiris, Slither, SmartCheck are much faster tools that take between 5 and 34
seconds on average to analyse a smart contract. HoneyBadger, Mythril, and Securify are slower and take
between 1m24s and 6m37s to execute. The difference in execution time between the tools is dependent
on the technique that each tool uses. Pure static analysis tools such as SmartCheck and Slither are really
fast since they only parse the AST of the contract to identify vulnerabilities and bad practices.
# Tools Average Total
1 Honeybadger 0:01:38 23 days, 13:40:002 Mythril 0:01:24 46 days, 07:46:553 Osiris 0:00:34 18 days, 10:19:014 Oyente 0:00:30 16 days, 04:50:115 Securify 0:06:37 217 days, 22:46:266 Slither 0:00:05 2 days, 15:09:367 SmartCheck 0:00:10 5 days, 12:33:14
Total 0:01:40 330 days and 15 hours
Table 4.7: Average execution time for each tool
Securify and Mythril analyse the EVM bytecode of the contracts. It means that those tools require
the contract to be compiled before doing the analysis. The additional compilation step slows down the
analysis.
The average execution time does not reflect the complete picture of the performance of a tool. Mythril,
for example, which has the best accuracy according to our evaluation, takes on average of 1m24s to analyse
a contract. It is faster than Securify that only has an accuracy of 9% compared to the 27% of Mythril.
52
4.2 Precision of the Static Analysis Tools
In this section we will extend the analysis regarding sbcurated done in Section 4.1.4. We manually analysed
each contract and each tool execution output to be able to compute precision metrics.
4.2.1 Testing Environment and Evaluation Criteria
For each tool output and contract analysed in Section 4.1.4, we manually verified the issues identified
by each tool on each contract and labeled them as either true positive (TP) or false positive (FP) after
performing a manual audit on the targeted contract to check the results. If the tool failed to detect a
vulnerability, it is flagged as a false negative (FN). The following metrics are used to evaluate the tools:
• True Positive (TP): Number of true positives, i.e. pieces of code which are indeed vulnerable and
the vulnerability is identified by the tool;
• False Positive (FP): Number of false positives, i.e. a vulnerability identified where it does not exist;
• False Negative (FN): Number of false negatives, i.e. the tool failed to identify a vulnerability.
Since the notions above defined do not apply equally to HoneyBadger, as it is a special case where a
true positive represents that the vulnerability does not exist (it detects honeypots) and their sample of
positives is not relevant (HoneyBadger had a total of five positives), we decided to not include it in this
discussion to avoid confusion.
Applying the defined notions we can introduce other useful measures to assess the performance of the
static analysis tools:
• False Negative Rate (FNR): FNR = FN / (FN + TP)
• False Discovery Rate (FDR): FDR = FP / (TP + FP)
• Sensitivity = 1 - FNR
• Precision = 1 - FDR
After carefully analysing all contracts for vulnerabilities, we identified a total of 167 vulnerabilities
in the 69 contracts of sbcurated. Table 4.8 reflects all the metrics of each tool regarding the execution of
the analysis discussed in Section 4.1.4. All informational outputs were not considered in the collection of
metrics. The same methodology followed in the analysis presented in the previous section was followed.
53
Mythril Osiris Oyente Securify Slither SmartCheck
TP 88 59 46 31 61 49FP 78 39 44 35 22 27FN 79 108 121 136 106 118FDR 47% 39,8% 48,9% 53,0% 26,5% 35,5%FNR 47,3% 64,7% 72,5% 81,4% 63,5% 70,7%Precision 53% 60,2% 51,1% 47% 73,5% 64,5%Sensitivity 52,7% 35,3% 27,5% 18,6% 36,5% 29,3%
Table 4.8: Metrics of the tools
It is possible to verify that the amount of positives (TP + FP) of Mythril and Securify are not
in line with the ones presented in Table 4.4. The reason is that Mythril usually outputs the same
vulnerability more than once, making the amount of positives in Table 4.4 bigger, as the analysis discussed
in Section 4.1.4 was automated and counts all the execution output. Table 4.8 does not take in account
repeated outputs, as it resulted from a manual analysis. For example, Mythril sometimes flagged a
vulnerable line and also flagged the function which that line was part of, repeating the vulnerability
without introducing new data. Regarding Securify, it usually repeats the same output in different types
of the same vulnerability. For example, it detects three types of Front Running (a.k.a Transaction
Order Dependence): TODReceiver, TODTransfer and TODAmount. In this context our goal is not to
distinguish the several types of Front Running and most Front Running positives given by Securify were
repeated three times through out this different types, i.e., most of the times Securify flags the same piece
of code vulnerable indicating the same line(s) in the three types of Front Running.
4.2.2 Summary
Mythril shows the best sensitivity, having the most number of true positives. As previously indicated by
Table 4.3, it is the one that flags more positives. It also seems to have the most number of false positives.
Slither is the tool that presents best precision (73,5%), followed by SmartCheck.
54
5SmartCheck Extension
Contents
5.1 SmartCheck Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 Patterns Added . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4 Availabity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
55
56
The analysis presented in the previous Chapter showed us that there is room for improvement for
static analysis to detect more vulnerabilities. Bad Randomness was one of the categories that all of the
tools failed to detect. In this section, we propose to extended the tool SmartCheck [6] to enable the
detection of vulnerabilities related to Bad Randomness and improve detection of Time Manipulation and
Access Control.
5.1 SmartCheck Patterns
SmartCheck runs lexical and syntactical analysis on Solidity source code. It uses a custom Solidity gram-
mar to generate a XML parse tree as an intermediate representation (IR). SmartCheck detects vulnera-
bility patterns by using XPath patterns on the IR. Our approach to improve SmartCheck vulnerabilities
detection is to investigate new rules and add them, in the form of XPath patterns.
Let us consider as an example the balance equality issue (an adversary can manipulate contract logic
by forcibly sending it ether) shown below in Listing 5.1, and also presented in the SmartCheck [6] paper.
The parse tree of the example is shown in Figure 5.1.
Listing 5.1: Balance equality.
1 if (this.balance == 42 ether) {...}
Figure 5.1: Parse tree for the Balance equality code example (SmartCheck, 2018)
To easily identify patterns of balance equality, the XPath pattern constructed in Listing 5.2 is suffi-
cient. As this is a relatively simple and straightforward example, it’s easy to precisely define the pattern
57
in a way that does not generate false positives.
Listing 5.2: XPath pattern for the Balance equality issue.
1 //expression[expression//envVarDef
2 [matches(text()[1],"ˆthis.balance$")]]
3 [matches(text()[1],"ˆ==|!=$")]
One of the problems of the SmartCheck approach, using XPath patterns, is that more complex rules
can not be precisely described using XPath. So, when more complex cases are needed to be put in form
of XPath patterns they can easily lead to false positives.
5.2 Patterns Added
We propose to add three more rules to SmartCheck and improve an already established rule.
Our first approach, based on the analysis presented in Chapter 4, was to add a rule to detect Bad Ran-
domness issues called SOLIDITY BAD RANDOMNESS. To do that, we created a XPath pattern to de-
tect the use of environment variables such as: block.number, block.coinbase, block.difficulty, block.gaslimit,
blockhash and block.blockhash. The XPath pattern is shown is Listing 5.3. This rule is straightforward
and its goal is to simply detect the use of the referred environment variables and to flag their use, acting
as a warning.
Listing 5.3: SOLIDITY BAD RANDOMNESS : XPath pattern to detect bad randomness vulnerabilities.
1 // expr e s s i on
2 [ e xp r e s s i on / env i ronmenta lVar iab le
3 [ matches ( t ex t ( ) [ 1 ] , ” block.number | b l o c k . c o i n b a s e | b l o c k . d i f f i c u l t y
4 | b l o c k . g a s l i m i t | blockhash | b lock .b l ockhash ” ) ]
5 [ not ( ance s to r : : ∗ [ 4 ] [ s e l f : : f u n c t i o n C a l l ] ) ] ]
Regarding Access Control, SmartCheck only detects tx.origin issues. We added a pattern to search
for suicides (selfdestruct) and ownership transfers where the function misses proper protection. To do
that we constructed two rules patterns inside a single rule named SOLIDITY UNPROTECTED. To
detect unprotected ownership transfers we created a pattern to look for all functions defined, excluding
constructors, that do not have standard modifiers defined, as onlyOwner, or require statements protecting
a value assignment to a variable defined as owner. To detect unprotected selfdestructs the approach was
similar, but instead we look for functions that do not have neither modifiers nor require statements
defined to protect selfdestruct calls. Since this a more complex approach and looks for standard names
used in Solidity programming, such as the variable owner to save the owner address or the modifier named
58
onlyOwner to protect sensitive calls, it is prone to detect more false positives. Both XPath patterns are
shown is Listing 5.4 and Listing 5.5, respectively.
Listing 5.4: SOLIDITY UNPROTECTED : XPath pattern to detect unprotected ownership transfers.
1 // f u n c t i o n D e f i n i t i o n [ not ( t ex t ( ) [ 1 ]=” cons t ruc to r ” ) ]
2 [ parameterLi s t / parameter ]
3 [ not ( boolean ( i d e n t i f i e r [ p o s i t i o n () >1])
4 or i d e n t i f i e r [ p o s i t i o n ( ) >1 ] [ matches ( t ex t ( ) [ 1 ] , ”onlyOwner | onlyOwner ( )
5 | onlyWal let | onlyWal let ( ) ” ) ]
6 or descendant : : e xp r e s s i on [ comparison [ t ex t () [1 ]=”==”] ]
7 [ e xp r e s s i on / env i ronmenta lVar iab le [ matches ( t ex t ( ) [ 1 ] , ” msg.sender ” ) ] ]
8 [ e xp r e s s i on / pr imaryExpress ion / i d e n t i f i e r
9 [ t ex t ( ) [ 1 ] = ( ance s to r : : c o n t r a c t D e f i n i t i o n // s t a t e V a r i a b l e D e c l a r a t i o n
10 [ typeName/elementaryTypeName [ t ex t ( ) [ 1 ]=” address ” ] ] / i d e n t i f i e r ) ] ] )
11 and descendant : : pr imaryExpress ion / i d e n t i f i e r [ t ex t ( ) = ”owner ” ] ]
Listing 5.5: SOLIDITY UNPROTECTED : XPath pattern to detect unprotected suicides.
1 // f u n c t i o n D e f i n i t i o n
2 [ not ( boolean ( i d e n t i f i e r [ p o s i t i o n () >1])
3 or i d e n t i f i e r [ p o s i t i o n ( ) >1 ] [ matches ( t ex t ( ) [ 1 ] , ”onlyOwner | onlyOwner ( )
4 | onlyWal let | onlyWal let ( ) ” ) ]
5 or descendant : : e xp r e s s i on [ comparison [ t ex t () [1 ]=”==”] ]
6 [ e xp r e s s i on / env i ronmenta lVar iab le [ matches ( t ex t ( ) [ 1 ] , ” msg.sender ” ) ] ]
7 [ e xp r e s s i on / pr imaryExpress ion / i d e n t i f i e r
8 [ t ex t ( ) [ 1 ] = ( ance s to r : : c o n t r a c t D e f i n i t i o n // s t a t e V a r i a b l e D e c l a r a t i o n
9 [ typeName/elementaryTypeName [ t ex t ( ) [ 1 ]=” address ” ] ] / i d e n t i f i e r ) ] ] )
10 and descendant : : f u n c t i o n C a l l // i d e n t i f i e r [ t ex t ( ) [ 1 ]=” s e l f d e s t r u c t ” ] ]
Regarding Time Manipulation, SmartCheck already defines a rule named SOLIDITY EXACT TIME,
where the pattern is defined to look for expressions that contain comparisons using block.timestamp or
now, except for cases when timestamp is a function argument. Analysing Table 4.3 we can see that
SmartCheck only detects one out of five Time Manipulation issues. We decided to improve this rule by
following a similar approach to the one used when creating our pattern to find Bad Randomness issues.
We decided to update the pattern to look for expressions that contains block.timestamp or now, not only
when used in comparisons as previously defined. The XPath pattern is detailed in Listing 5.6. The design
approach was the same as in the rule constructed to detect Bad Randomness issues: to flag the use of
variables related to Time Manipulation, not only in comparisons, and thus act as a warning.
59
Listing 5.6: SOLIDITY EXACT TIME : XPath pattern to detect time manipulation vulnerabilities.
1 // expr e s s i on
2 [ e xp r e s s i on / env i ronmenta lVar iab le
3 [ matches ( t ex t ( ) [ 1 ] , ” ˆ block.t imestamp | now$ ” ) ]
4 [ not ( ance s to r : : ∗ [ 4 ] [ s e l f : : f u n c t i o n C a l l ] ) ] ]
5.3 Results
We analysed sbcurated with SmartCheck Extended, using the same version used in the study presented in
Chapter 4, to set a fair ground of comparison. Table 5.1 compares the results of SmartCheck, discussed
in Section 4.1.4, and the results obtained from executing SmartCheck Extended in the same dataset,
sbcurated. We can observe that our extension is capable of detecting a total of 15 more issues, with
regard of the annotated vulnerabilities in sbcurated, more than doubling the capability of detection of
SmartCheck without our extension. With our proposed extension we can detect 24% of the vulnerabilities
annotated in sbcurated, instead of the previous 11%.
Category SmartCheck SmartCheck Extended
Access Control 2/19 11% 4/19 21%Arithmetic 1/22 5% 1/22 5%Bad Randomness 0/31 5% 10/31 32%Denial of Service 0/7 0% 0/7 0%Front Running 0/7 0% 0/7 0%Reentrancy 5/8 62% 5/8 62%Short Addresses 0/1 0% 0/1 0%Time Manipulation 1/5 20% 4/ 5 80%Unchecked Low Level Calls 4/12 33% 4/12 33%Other 0/3 0% 0/3 0%
Total 13/115 11% . 28/115 24%
Table 5.1: Vulnerabilities identified per category by SmartCheck and SmartCheck Extended in sbcurated.
When taking in consideration the precision metrics presented in Section 4.2, where all 167 vulnera-
bilities present in sbcurated are taken into account, our solution improves 15,1% in sensitivity and 5,1%
in precision, as seen in Table 5.2. We can see, as previously indicated and suspected, that the Access
Control rule has a considerable amount of false positives, actually more than the double of true positives.
Regarding the rules introduced to detect Bad Randomness and Time Manipulation, their result is as
expected, since they act as warnings and detect the use of sensitive environment variables.
60
SmartCheck SmartCheck Extended
TP 49 74FP 27 32FN 118 93FDR 35,5% 30,2%FNR 70,7% 55,6%Precision 64,5% 69,6%Sensitivity 29,3% 44,4%
Table 5.2: SmartCheck and SmartCheck Extended metrics.
Access Control Bad Randomness Time Manipulation
TP 2 10 13FP 5 - -Warnings - 37 4
Table 5.3: Positives of SmartCheck Extended.
Table 5.3 divides all the vulnerabilities detected by our extension in TP, FP or warnings. True
positives map the positives flagged from the 167 vulnerabilities taken in consideration in Section 4.2.
False positives are the number of issues flagged that are not true. Warnings are the number of issues
detected that are not mapped in the 167 vulnerabilities, but also are not false positives. As previous
stated, SOLIDITY BAD RANDOMNESS and SOLIDITY EXACT TIME are meant to act as warnings
and simply flag the use of sensitive environment variables.
5.4 Availabity
SmartCheck Extended is available on GitHub1, as a fork of the original SmartCheck. It is also included
in SmartBugs and ready to be executed.
1SmartCheck Extended: https://github.com/pedrocrvz/smartcheck
61
62
6Conclusion
Contents
6.1 Achievements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
63
64
In this chapter we describe the main contributions of this dissertation while assessing if our objectives
were met. We also discuss future work and improvements.
6.1 Achievements
As smart contracts use grows, more vulnerabilities are found and their impact becomes bigger. With
our work we contributed to improve the research on static analysis tools and introduced new tools to
facilitate research and automated analysis.
We described the most important vulnerabilities found in Ethereum smart contracts, using DASP 10
as taxonomy, and enumerated the state-of-the-art static analysis tools available to verify smart contracts.
After our literature review we introduced SmartBugs, an important contribution to automate and ease
the execution of smart contracts analysers. We also introduced sbcurated, a dataset of 143 annotated
vulnerable Solidity smart contracts with a defined vulnerability taxonomy that can serve as a reference
dataset to the research community. We referenced sbwild, used in [7] and discussed in Chapter 4.1. Both
datasets are publicly available for the community to use.
Using SmartBugs, we were able to organize and easily execute 7 state-of-the-art tools to perform
an empirical review on 47,587 Ethereum smart contracts. We discussed the results and obtained useful
remarks and insights. For example, we were able to detect that Bad Randomness is a category of
vulnerabilities overlooked by the 7 tools reviewed. This motivated us to contribute to the field by
extending SmartCheck to detect Bad Randomness vulnerabilities. Our extension improved SmartCheck
by 15,1% in sensitivity and 5,1% in precision.
With our work we accomplished the objectives proposed and all the topics mentioned in the research
questions were addressed. We are now able to provide a direct and succint answer to all the research
questions:
What are the state-of-the-art analysis tools for Ethereum smart contracts?
In our research we listed 35 state-of-the-art tools currently available to analyse Ethereum smart contracts.
Furthermore, we listed all their URLs and, where applicable, we identified the corresponding research
paper. We also reviewed in detail 7 state-of-the-art tools.
What is the precision of the state-of-the-art analysis tools in detecting vulnerabilities on
Ethereum smart contracts? In Table 4.8 we established the precision and sensitivity of six state-of-
the-art tools. Slither showed to be the most precise and Mythril seems to have the highest sensitivity.
Which categories of vulnerabilities are most detected? Table 4.3 shows vulnerabilities identified
per category by each tool. The categories Other, Reentrancy, Arithmetic and Unchecked Low Level Calls
65
are the most detected. Table 4.6 also reiterates that Other, Arithmetic, Reentrancy and Unchecked Low
Level Calls are the categories of vulnerabilities most detected. With this data we can also say that they
seem to be the most common type of vulnerabilities.
How many vulnerable contracts are present in the Ethereum blockchain? The seven tools
identified vulnerabilities in 93% of the contracts (in 44,589 contracts), which suggests a high number of
false positives. Oyente, alone, detects vulnerabilities in 73% of the contracts. By combining the tools to
create a consensus, we observe that only a few number of vulnerabilities received a consensus of four or
more tools: 937 for Arithmetic and 133 for Reentrancy. Unchecked Low Level Calls receives consensus in
52 vulnerabilities, Denial Of Service and Other in 50 and 41, respectively.
How long do the tools require to analyse the smart contracts? On average, the tools take 1m40s
to analyse one contract. Slither is the fastest tool and takes on average only 5 seconds to analyse a
contract. We also observe that the execution speed is not the only factor that impacts the performance of
the tools. Securify was the slowest tool on average, but Securify can easily be parallelized and therefore
analyse the 47,518 smart contracts faster. Finally, we have not observed a correlation between accuracy
and execution time.
How can we improve a state-of-the-art analysis tool? The results gathered in the empirical review
discussed in Chapter 4 showed us that Bad Randomness was one category of vulnerabilities overlooked
by the 7 tools studied. By presenting an extension of SmartCheck, introducing the capability to detect
Bad Randomness issues, in Chapter 5, we already contributed to improve a state-of-the-art analysis tool.
Further more, there is still room to improve the tools studied by increasing the scope of categories of
vulnerabilities detected and enrich the quality of the analysis, decreasing the number of false positives
and improving accuracy.
6.2 Future Work
As discussed throughout this work, static analysis of smart contracts has been receiving several contri-
butions in recent years. As blockchain and smart contracts have a fast pace of development, we should
aim to keep our work up to date. Some suggestions for future work include:
• Improve and keep SmartBugs updated: SmartBugs has room for improvements and more
tools should be added. SmartBugs WUI should have a better error handling mechanism and we
can still provide better insights about the vulnerabilities detected, adding a better way to show the
output to the user.
66
• Deploy SmartBugs: SmartBugs WUI can be deployed as a website to be easily accessible to
the community, providing a way for developers to have access to multiple tools to test their smart
contracts using SmartBugs to automate the process.
• Update sbcurated: The dataset can grow and should be constantly updated in future to have more
contracts or even support new types of vulnerabilities. Contracts with same vulnerabilities, flagged
by 4 or more tools in the analysis done in 4.1.5 using the dataset sbwild, hint consensus on the
vulnerability and thus suggest a true positive. For example, 937 Arithmetic vulnerabilities received
a consensus of four or more tools. A manual audit of these contracts can be performed to confirm it
and, if the vulnerability exists, the contract should be properly annotated in order to be added to
sbcurated. Some of this work is already in progress, some of the contracts were already added, and
our next goal, along with improving SmartBugs WUI, is to keep updating and improving sbcurated
by adding such contracts.
• Improve SmartCheck Extension: SmartCheck Extension can be further extended by adding
new XPtah patterns to improve its capability to detect vulnerabilities on Ethereum smart contracts.
For example, SmartCheck still overlooks Denial of Service and Front Running vulnerabilities and
as poor detection capability regarding Arithmetic issues.
67
68
Bibliography
[1] N. Atzei, M. Bartoletti, and T. Cimoli, “A survey of attacks on ethereum smart contracts (sok),” in
Principles of Security and Trust. Springer, 2017, pp. 164–186.
[2] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” https:// bitcoin.org/ bitcoin.pdf ,
2008.
[3] G. Wood et al., “Ethereum: A secure decentralised generalised transaction ledger,” Ethereum project
yellow paper, vol. 151, no. 2014, pp. 1–32, 2014.
[4] I. Nikolic, A. Kolluri, I. Sergey, P. Saxena, and A. Hobor, “Finding the greedy, prodigal, and suicidal
contracts at scale,” in Proceedings of the 34th Annual Computer Security Applications Conference.
New York, NY, USA: ACM, 2018, pp. 653–663.
[5] L. Luu, D.-H. Chu, H. Olickel, P. Saxena, and A. Hobor, “Making smart contracts smarter,” in
Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. New
York, NY, USA: ACM, 2016, pp. 254–269.
[6] S. Tikhomirov, E. Voskresenskaya, I. Ivanitskiy, R. Takhaviev, E. Marchenko, and Y. Alexandrov,
“Smartcheck: Static analysis of ethereum smart contracts,” in 2018 IEEE/ACM 1st International
Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB). Gothenburg,
Sweden, Sweden: IEEE, 2018, pp. 9–16.
[7] T. Durieux, J. F. Ferreira, R. Abreu, and P. Cruz, “Empirical review of automated analysis tools on
47,587 ethereum smart contracts,” 2019, https://arxiv.org/pdf/1910.10601.
[8] M. Romiti, A. Judmayer, A. Zamyatin, and B. Haslhofer, “A deep dive into bitcoin mining pools:
An empirical analysis of mining shares,” arXiv preprint arXiv:1905.05999, 2019.
[9] N. Szabo, “Smart contracts: Building blocks for digital markets,” 1996.
[10] Ethereum, “Proof of stake,” https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQs, 2018.
[11] NCCGroup, “Decentralized application security project (or dasp) top 10,” https://dasp.co, 2018.
69
[12] P. Daian, “Analysis of the dao exploit,” http://hackingdistributed.com/2016/06/18/
analysis-of-the-dao-exploit/, 2016.
[13] Solidity, “Solidity documentation,” https://solidity.readthedocs.io/en/v0.5.1/, 2018.
[14] A. van der Stock, B. Glas, and T. Gigler, “Owasp top 10 2017,” The Ten Most Critical Web Appli-
cation Security Risks, 2017.
[15] M. Suiche, “The $280m ethereum’s parity bug.” 2017, https://blog.comae.io/
the-280m-ethereums-bug-f28e5de43513.
[16] D. A. Manning, “Solidity security: Comprehensive list of known attack vectors and common anti-
patterns,” https://blog.sigmaprime.io/solidity-security.html, 2018.
[17] Trailofbits, “(not so) smart contracts,” https://github.com/trailofbits/not-so-smart-contracts, 2018.
[18] KingOfEther, “Post-mortem investigation (feb 2016),” https://www.kingoftheether.com/
postmortem.html.
[19] ConsenSys, “Ethereum smart contract security best practices,” https://consensys.github.io/
smart-contract-best-practices/.
[20] F. Memoria, “Smartbillions challenges hackers with 1,500 ether re-
ward, gets hacked and pulls most of it out,” https://www.ccn.com/
smartbillions-challenges-hackers-1500-ether-reward-gets-hacked-pulls/.
[21] D. K. Mikhail Vladimirov, “Erc20 api: An attack vector on approve/transferfrom methods.”
[22] I. Bogatyy, “Implementing ethereum trading front-runs on the bancor exchange in python,” https:
//hackernoon.com/front-running-bancor-in-150-lines-of-python-with-ethereum-api-d5e2bfd0d798.
[23] P. Bylica, “How to find $10m just by reading the blockchain,” https://blog.golemproject.net/
how-to-find-10m-by-just-reading-blockchain-6ae9d39fcd95.
[24] P. Cousot and R. Cousot, “Abstract interpretation: a unified lattice model for static analysis of
programs by construction or approximation of fixpoints,” in Proceedings of the 4th ACM SIGACT-
SIGPLAN symposium on Principles of programming languages. ACM, 1977, pp. 238–252.
[25] M. di Angelo and G. Salzer, “A survey of tools for analyzing ethereum smart contracts,” in 2019 IEEE
International Conference on Decentralized Applications and Infrastructures (DAPPCON). Newark,
CA, USA, USA: IEEE, April 2019, pp. 69–78.
70
[26] S. Azzopardi, J. Ellul, and G. J. Pace, “Monitoring smart contracts: Contractlarva and open chal-
lenges beyond,” in Runtime Verification, C. Colombo and M. Leucker, Eds. Cham: Springer
International Publishing, 2018, pp. 113–137.
[27] R. Norvill, B. B. F. Pontiveros, R. State, and A. Cullen, “Visual emulation for ethereum’s vir-
tual machine,” in NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium.
Taipei, Taiwan: IEEE, 2018, pp. 1–4.
[28] Y. Zhou, D. Kumar, S. Bakshi, J. Mason, A. Miller, and M. Bailey, “Erays: Reverse engineering
ethereum’s opaque smart contracts,” in 27th USENIX Security Symposium (USENIX Security
18). Baltimore, MD: USENIX Association, Aug. 2018, pp. 1371–1385. [Online]. Available:
https://www.usenix.org/conference/usenixsecurity18/presentation/zhou
[29] H. Liu, C. Liu, W. Zhao, Y. Jiang, and J. Sun, “S-gram: towards semantic-aware security auditing
for ethereum smart contracts,” in Proceedings of the 33rd ACM/IEEE International Conference on
Automated Software Engineering. New York, NY, USA: ACM, 2018, pp. 814–819.
[30] I. Grishchenko, M. Maffei, and C. Schneidewind, “A semantic framework for the security analysis
of ethereum smart contracts,” in Principles of Security and Trust, L. Bauer and R. Kusters, Eds.
Cham: Springer International Publishing, 2018, pp. 243–269.
[31] E. Albert, P. Gordillo, B. Livshits, A. Rubio, and I. Sergey, “Ethir: A framework for high-level
analysis of ethereum bytecode,” in Automated Technology for Verification and Analysis, S. K. Lahiri
and C. Wang, Eds. Cham: Springer International Publishing, 2018, pp. 513–520.
[32] A. Mavridou and A. Laszka, “Tool demonstration: Fsolidm for designing secure ethereum smart
contracts,” in Principles of Security and Trust, L. Bauer and R. Kusters, Eds. Cham: Springer
International Publishing, 2018, pp. 270–277.
[33] T. Chen, X. Li, X. Luo, and X. Zhang, “Under-optimized smart contracts devour your money,”
in 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering
(SANER). Klagenfurt, Austria: IEEE, 2017, pp. 442–446.
[34] C. F. Torres, M. Steichen, and R. State, “The art of the scam: Demystifying honeypots
in ethereum smart contracts,” in 28th USENIX Security Symposium (USENIX Security
19). Santa Clara, CA: USENIX Association, Aug. 2019, pp. 1591–1607. [Online]. Available:
https://www.usenix.org/conference/usenixsecurity19/presentation/ferreira
[35] E. Hildenbrandt, M. Saxena, N. Rodrigues, X. Zhu, P. Daian, D. Guth, B. Moore, D. Park, Y. Zhang,
A. Stefanescu et al., “Kevm: A complete formal semantics of the ethereum virtual machine,” in 2018
71
IEEE 31st Computer Security Foundations Symposium (CSF). Oxford, UK: IEEE, 2018, pp. 204–
217.
[36] N. Grech, M. Kong, A. Jurisevic, L. Brent, B. Scholz, and Y. Smaragdakis, “Madmax: Surviving out-
of-gas conditions in ethereum smart contracts,” Proceedings of the ACM on Programming Languages,
vol. 2, no. OOPSLA, p. 116, 2018.
[37] M. Mossberg, F. Manzano, E. Hennenfent, A. Groce, G. Grieco, J. Feist, T. Brunson, and
A. Dinaburg, “Manticore: A user-friendly symbolic execution framework for binaries and smart
contracts,” 2019.
[38] B. Mueller, “Smashing ethereum smart contracts for fun and real profit,” in 9th Annual HITB
Security Conference (HITBSecConf). Amsterdam, Netherlands: HITB, 2018.
[39] C. F. Torres, J. Schutte et al., “Osiris: Hunting for integer bugs in ethereum smart contracts,” in
Proceedings of the 34th Annual Computer Security Applications Conference. New York, NY, USA:
ACM, 2018, pp. 664–676.
[40] M. Suiche, “Porosity: A decompiler for blockchain-based smart contracts bytecode,” DEF con,
vol. 25, p. 11, 2017.
[41] C. Liu, H. Liu, Z. Cao, Z. Chen, B. Chen, and B. Roscoe, “Reguard: finding reentrancy bugs
in smart contracts,” in Proceedings of the 40th International Conference on Software Engineering:
Companion Proceeedings. New York, NY, USA: ACM, 2018, pp. 65–68.
[42] E. Zhou, S. Hua, B. Pi, J. Sun, Y. Nomura, K. Yamashita, and H. Kurihara, “Security assurance
for smart contract,” in 2018 9th IFIP International Conference on New Technologies, Mobility and
Security (NTMS). Paris, France: IEEE, Feb 2018, pp. 1–5.
[43] J. Chang, B. Gao, H. Xiao, J. Sun, and Z. Yang, “scompile: Critical path identification and analysis
for smart contracts,” 2018.
[44] P. Tsankov, A. Dan, D. Drachsler-Cohen, A. Gervais, F. Buenzli, and M. Vechev, “Securify: Prac-
tical security analysis of smart contracts,” in Proceedings of the 2018 ACM SIGSAC Conference on
Computer and Communications Security. New York, NY, USA: ACM, 2018, pp. 67–82.
[45] J. Feist, G. Greico, and A. Groce, “Slither: A static analysis framework for smart contracts,”
in Proceedings of the 2Nd International Workshop on Emerging Trends in Software Engineering
for Blockchain, ser. WETSEB ’19. Piscataway, NJ, USA: IEEE Press, 2019, pp. 8–15. [Online].
Available: https://doi.org/10.1109/WETSEB.2019.00008
72
[46] P. Hegedus, “Towards analyzing the complexity landscape of solidity based ethereum smart con-
tracts,” Technologies, vol. 7, no. 1, p. 6, 2019.
[47] J. Krupp and C. Rossow, “teether: Gnawing at ethereum to automatically exploit smart
contracts,” in 27th USENIX Security Symposium (USENIX Security 18). Baltimore, MD: USENIX
Association, Aug. 2018, pp. 1317–1333. [Online]. Available: https://www.usenix.org/conference/
usenixsecurity18/presentation/krupp
[48] L. Brent, A. Jurisevic, M. Kong, E. Liu, F. Gauthier, V. Gramoli, R. Holz, and B. Scholz, “Vandal:
A scalable security analysis framework for smart contracts,” 2018.
[49] S. K. Lahiri, S. Chen, Y. Wang, and I. Dillig, “Formal specification and verification of smart contracts
for azure blockchain,” 2018.
[50] S. Kalra, S. Goel, M. Dhawan, and S. Sharma, “ZEUS: analyzing safety of smart contracts,” in 25th
Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California,
USA, February 18-21, 2018. San Diego, California, USA: NDSS, 2018, pp. 1–15.
[51] I. Grishchenko, M. Maffei, and C. Schneidewind, “A semantic framework for the security analysis
of ethereum smart contracts,” in Principles of Security and Trust, L. Bauer and R. Kusters, Eds.
Cham: Springer International Publishing, 2018, pp. 243–269.
[52] D. Perez and B. Livshits, “Smart contract vulnerabilities: Does anyone care?” 2019.
[53] R. M. Parizi, A. Dehghantanha, K.-K. R. Choo, and A. Singh, “Empirical vulnerability
analysis of automated smart contracts security testing on blockchains,” in Proceedings of
the 28th Annual International Conference on Computer Science and Software Engineering,
ser. CASCON ’18. Riverton, NJ, USA: IBM Corp., 2018, pp. 103–113. [Online]. Available:
http://dl.acm.org/citation.cfm?id=3291291.3291303
73
74
75