irma documentation - media. · pdf fileirma documentation, release 2.0.4 irma is an...

IRMA DocumentationRelease 2.0.4

Quarkslab

Mar 13, 2018

Contents

1 Introduction 31.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 File Analysis Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Infrastructure Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Hardware requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.5 Supported Analyzers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Automated Install 92.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Ansible scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Predefined Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Using Debian repos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Manual Installation 133.1 Brain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3 Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 Database migration 414.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.2 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.3 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.4 Tips and tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5 To evolve IRMA 475.1 Adding a new probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6 References 516.1 Disclaimer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.2 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.3 Apache License, version 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.4 Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7 Frequently Asked Questions 537.1 Playing with tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537.2 SSL settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567.3 How to debug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

i

7.4 How to migrate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607.5 API documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617.6 Connect to a vagrant box through ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.7 Enable SSL using OpenSSL in ansible scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.8 Speed up your Vagrant VMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8 Resources 65

9 Screenshots 679.1 Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679.2 Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

ii

IRMA Documentation, Release 2.0.4

IRMA is an asynchronous & customizable analysis system for suspicious files.

This guide will explain how to set up IRMA, use it and customize it at will.

Contents 1


2 Contents

CHAPTER 1

Introduction

This is an introductory chapter to IRMA. It recalls IRMA’s overall architecture, hardware required to run it and therecommended order for installing the IRMA’s components.

1.1 Purpose

IRMA intends to be an open-source platform designed to help identifying and analyzing malicious files.

However, today’s defense is not only about learning about a file, but it is also getting a fine overview of the incidentyou dealt with: where / when a malicious file has been seen, who submitted a hash, where a hash has been noticed,which anti-virus detects it, . . .

An important value with IRMA comes from you keep control over where goes and who gets your data. Once youinstall IRMA on your network, your data stays on your network.

Each submitted files is analyzed in various ways. For now, we focus our efforts on multiple anti-virus engines, but weare working on other “probes” (feel free to submit your own).

1.2 File Analysis Process

1. An analysis begins when a user uploads files to the Frontend.

2. Frontend checks for existing files and results in SQL. If needed, it stores the new files and calls asynchronouslyscan jobs on Brain.

3. Brain worker sends as much subtasks to Probe(s) as needed.

4. Probe workers process their jobs and send back results to Brain.

5. Brain sends results to Frontend.

3


1.3 Infrastructure Overview

A drawing is better than a lot of explanations (sometimes ;)

4 Chapter 1. Introduction


1.4 Hardware requirements

IRMA platform is divided in three major components: the Frontend, the Brain and one or multiple Probes.

These three components can be installed on a unique host or on multiple hosts, according to the kind of probes youare using.

The Frontend and the Brain must be installed on a GNU/Linux system1. We recommend to use a Debian Stabledistribution which is supported and known to work.

According to the kind of probes and their dependencies, each analyzers can be installed on a separate hosts or share

1 Theorically, it should be possible, with some efforts, to make IRMA work on Microsoft Windows systems as most of the components used forthe platform are known to work or to have equivalents on these systems.

1.4. Hardware requirements 5


the same host as far as they do not interfere with each other2. So forth, only Debian Stable and Microsoft Windows 8and 10 hosts have been tested.

We can not give you any specific numbers. On one hand we managed to run the whole IRMA platform on a singlemachine by hosting it with multiple systems inside virtual machines: this setup gives fairly high throughput as longas it has reasonable IO (ideally, SSDs), and a good amount of memory (our setup was an i7 cpu with 16 GB ramon regular drives (at least 200 GB required), on the other hand, a lighter version of the system with the three partstogether3 was successfully installed on a single virtual machine (1 GB of Ram and 4 virtual processors).

For a large company, in theory, given a single high-memory machine, with 16+ cores, and SSDs, you could run IRMAplatform and bear the workload load with reasonable response time.

1.5 Supported Analyzers

We have mainly focused our efforts on multiple anti-virus engines but we are working on other kind of “probes”. Weenumerate the analyzers that are bundled with IRMA probe application. Feel free to submit your own probes.

1.5.1 Antiviruses

So far, we have instrumented the following antiviruses from their CLI:

Probe Name Anti-Virus Name PlatformASquaredCmd Emsisoft Command Line Microsoft Windows CLIAvira Avira Microsoft Windows CLIAvastCoreSecurity Avast GNU/Linux CLIAVGAntiVirusFree AVG GNU/Linux CLIBitdefenderForUnices Bitdefender GNU/Linux CLIClamAV ClamAV GNU/Linux CLIComodoCAVL Comodo Antivirus for Linux GNU/Linux CLIDrWeb Dr.Web GNU/Linux CLIEsetNod32 Eset Nod32 Business Edition GNU/Linux CLIEScan eScan GNU/Linux CLIFProt F-Prot GNU/Linux CLIFSecure F-Secure GNU/Linux CLIGData G Data Antivirus Microsoft Windows CLIKaspersky Kaspersky Internet Security Microsoft Windows CLIMcAfeeVSCL McAfee VirusScan Command Line GNU/Linux - Microsoft Windows CLISophos Sophos GNU/Linux - Microsoft Windows CLISymantec Symantec Endpoint Protection Microsoft Windows CLIVirusBlokAda VirusBlokAda GNU/Linux CLIZoner Zoner Antivirus GNU/Linux CLI

1.5.2 External analysis platforms

So far, we query the following external analysis platforms:

2 For instance, we managed to host several GNU/Linux anti-viruses on an unique probe by preventing it to launch daemons at startup. This isdifficult for Microsoft systems on which it is not recommended to install multiple anti-viruses on a single host.

3 with a limited set of probes



Probe Name Analysis Platform DescriptionVirusTotal VirusTotal Report is searched using the sha256 of the file which is not sent

1.5.3 File database

So far, we query the following file databases:

ProbeName

Database Description

NSRL National Software Reference Li-brary

collection of digital signatures of known, traceable software ap-plications

1.5.4 Metadata

So far, we implemented the following analyzers:

Probe Name DescriptionStaticAnalyzer PE File analyzer adapted from Cuckoo SandboxPEiD PE File packer analyzerYara Checks if a file match yara rules

1.5. Supported Analyzers 7

CHAPTER 2

Automated Install

IRMA platform can be easily installed with a set of ansible roles and playbooks. It will help you to build, install ormaintain different setups.

2.1 Requirements

• Ansible 2.2.1.0;

2.2 Ansible scripts

Get IRMA ansible scripts on github:

$ git clone https://github.com/quarkslab/irma$ cd irma/ansible

2.3 Predefined Environments

There are 2 different IRMA setups available. Dev/Testing will be installed in one or multiple virtual machines whileproduction could be used to install IRMA on physical machines or virtual machines already setup:

2.3.1 Development environment

This environment has been designed to help you to modify IRMA’s components and redeploy and test them. In thissetup, everything is installed in a single virtual machine with sources rsync-ed between the host and the guest.

9

http://www.ansible.com

https://github.com/ansible/ansible


Requirements

• Vagrant 1.8 or higher has to be installed

• As the installation work only for Virtualbox, you will need to install it

• Rsync to synchronize directories from host to VMs

• Read the Ansible introduction

Run Vagrant and create your VMs

To initialize and provision the Virtualbox VM.

$ cd <IRMA_SRC_DIR>/ansible$ VM_ENV=your_environment_name vagrant up

The template will be downloaded automatically and configured using environments/dev.yml file.

Note: Optionally, if you want to use your own environment, create it in environments directory and run:

$ VM_ENV=your_environment_name vagrant up

Configure your .ini files

Note: You can bypass this step, as this provisioning is sync with default username and password used in (fron-tend|brain|probe) config files.

As your config/*.ini file are transferred from host to VMs, you will need to modify them individually in thefrontend, probe and brain directories to match the user and password defined in playbooks/group_vars/*.

Modify your host and open IRMA frontend

Then, for proper use, update your /etc/hosts file and add:

172.16.1.30 www.frontend.irma

Then, with your web browser, IRMA allinone is available at www.frontend.irma.

Sync files between host and guest

Once rsync is installed inside your virtual machine and your environment is correctly set. You could easily sync yourcode with:

$ vagrant rsync # or vagrant rsync-auto to automatically initiates an rsync# transfer when changes are detected

Then, reload the modified application.

10 Chapter 2. Automated Install

http://www.vagrantup.com/

https://www.virtualbox.org/

https://rsync.samba.org/

http://docs.ansible.com/intro.html

http://www.frontend.irma


2.3.2 Production environment

This environment is used to install IRMA on production-ready Debian servers.

Requirements

• One or multiple 64-bit Debian 7 servers.

Preparing servers

Create an account that is going to be used to provision IRMA on the server via Ansible, or use one which has alreadybeen created. To speed up provisioning, you can:

• Authorize your SSH key for password-less authentication (optional):

# On your local machine$ ssh-copy-id user@hostname # -i if you want to select your identity file

• If you do not want to have to type your password for sudo command, consider adding your user to sudoers,using visudo command (optional):

user ALL=(ALL) NOPASSWD: ALL

Configure for your installation

Modify settings in playbooks/group_vars/all especially the default_ssh_keys: section. You will needto add your public keys for SSH password-less connection to the default irma server user.

Configuration file used by the brain, the frontend and the probes applications are generated with default values that arespecified in playbooks/group_vars/brain, playbooks/group_vars/frontend and playbooks/group_vars/probe respectively. Make sure to adapt xxx_deployment_configs variables accordingly toyour installation. It is recommended to change all the default passwords defined in group_vars/* configurationfiles (password variables for most of them).

Finally, you will need to customize the hosts/example file and adapt it to describe your own server infrastruc-ture. There is three sections, one for each server role (frontend, brain, probe). Please refer to Ansible Inventorydocumentation for the expected syntax.

Run the Ansible Playbook

To run the main playbook with the hosts/example file you have defined, use the following command. Ansiblewill ask you the sudo password (-K option).

$ cd <IRMA_SRC_DIR>/ansible$ ansible-playbook -i ./hosts/example playbooks/playbook.yml -u <your_sudo_username> -→˓K

To run one or more specific actions and avoid running all the playbook, you can use tags. For example, if you want tore-provision Nginx, run the same command, but append --tags=nginx. You can combine multiple tags separatedwith commas.

2.3. Predefined Environments 11

https://www.debian.org

http://docs.ansible.com/intro_inventory.html#inventory

http://docs.ansible.com/intro_inventory.html#inventory


Deploy a new version of IRMA

Assuming that you have already provisioned and deployed a version of IRMA, which you want to upgrade, you willneed to run the deployment script:

$ ansible-playbook -i ./hosts/example ./playbooks/deployment.yml -u irma

Make sure to replace irma with the default user if you have changed it in the group_vars/all file.

Access your IRMA installation

Once the provisioning and the deployment have finished, you should be able to perform scans and get their results viathe frontend hostname you specified.

2.4 Using Debian repos

If you planned to use only Debian official repository, you’ll need to change in playbooks/group_vars/all:

default_use_debian_repo: yes # no is the default value

12 Chapter 2. Automated Install

CHAPTER 3

Manual Installation

Warning: Manual installation needs an update since 1.5.3

as all repositories layout have changed. Try Automated install or ask for help on IRC.

In some specific cases where automatic install is not possible, you should go with manual installation.

Each major component of the IRMA platform comes with their own python-based application. As the Brain is thenerve center of the whole platform, it is recommended to install it first before installing other components. One canthen install either the Frontend or the Probes he wants.

Please refer to specific documentation for the manual installation process for each component.

3.1 Brain

The Brain is a python-based application that only dispatches analysis requests from different frontends1 to the avail-able Probes. Analyses are scheduled by the Brain on Probes through Celery, an open source task framework forPython.

3.1.1 Architecture

Let us recall first the inner architecture of the Brain. It uses multiple technologies with a specific purpose each:

• a Celery worker that handles scan requests from Frontends and results returned by the Probes.

• a RabbitMQ server used by Celery as a backend and as a broker for task queues and job queues used to scheduletasks for Probes (for scan jobs) and the Frontend (for scan results).

• an SFTP server where files to be scanned are uploaded by Frontends and downloaded by Probes,

1 This feature is not ready yet, we are currently working on its implementation.

13


3.1.2 Installing and Configuring RabbitMQ

The following explains how to install and to setup RabbitMQ for your setup.

Install RabbitMQ

RabbitMQ server can be installed with this command on Debian:

$ sudo apt-get install rabbitmq-server

Configuring RabbitMQ

RabbitMQ serves all components of IRMA platform. Each component has their own virtual host (i.e., message queuewhere to get data), a username and a password.

To configure RabbitMQ for IRMA platform, you have to create users, vhosts and add permissions for each user tocorresponding vhost. To easily create virtual hosts and users, we provide scripts in extras/ directory:

$ sh ./extras/scripts/rabbitmq/rmq_adduser.shUsage: sudo rmq_adduser.sh <user> <password> <vhost>

For instance, to create 3 users with the following parameters, one can do:

User-name

Password VirtualHost

Command

brain brain-rmq-password

mqbrain sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brainbrain-rmq-password mqbrain

probe probe-rmq-password

mqprobe sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probeprobe-rmq-password mqprobe

fron-tend

frontend-rmq-password

mqfron-tend

sudo ./extras/scripts/rabbitmq/rmq_adduser.shfrontend frontend-rmq-password mqfrontend

The script simply execute the following three commands:

$ sudo rabbitmqctl add_user <username> <password>$ sudo rabbitmqctl add_vhost <vhostname>$ sudo rabbitmqctl set_permissions -p <vhostname> <username> ".*" ".*" ".*"

Warning: Important

Make sure to note down the username, the password and the virtual host you just defined. You will be asked toretype them on each application configuration file (brain, frontend and probe)

Note: Disclaimer

Please ensure that only trusted sources can communicate with your RabbitMQ server, by setting up firewall rules forinstance, as your RabbitMQ may be exposed to Internet.

14 Chapter 3. Manual Installation


Verifying RabbitMQ configuration

We can verify that the RabbitMQ server has taken into account our modifications with some commands:

Checking for vhosts

$ sudo rabbitmqctl list_vhostsListing vhosts ...mqbrain/mqfrontendmqprobemqadmin...done.

If the defined virtual host are not listed by the above command, please execute once more the script.

Checking for users

$ sudo rabbitmqctl list_usersListing users ...probe []brain []frontend []...done.

If the defined users are not listed by the above command, please execute once more the script.

Changing password

If you do not remember the password you just typed, you can change it with rabbitmqctl command:

$ sudo rabbitmqctl change_password brain brain-rmq-passwordChanging password for user "brain" ......done.

Restarting the service

You may want to restart the service. Thus, the following command can be done:

$ sudo invoke-rc.d rabbitmq-server restart

3.1.3 Installing and configuring sftp

A SFTP server is used to transfer files Frontends and meant to be analyzed by Probes. We describe in the followinghow to set it up.

3.1. Brain 15


Installing sshd

sshd should already been installed but if not install it with:

$ sudo apt-get install openssh-sftp-server

Creating FTP specific users and groups

$ sudo groupadd sftpusers$ sudo useradd -g sftpusers -M -s /etc <frontend username>$ sudo useradd -g sftpusers -Match -s /etc <probe username>$ sudo passwd frontend

<enter frontend password>$ sudo passwd probe

<enter probe password>

Configure sshd

Edit /etc/sshd_config

Subsystem sftp internal-sftpMatch User <frontend username>

AllowTcpForwarding noChrootDirectory /sftp/<frontend username>ForceCommand internal-sftp -u 2PermitTunnel noX11Forwarding no

Match User <probe username>AllowTcpForwarding noChrootDirectory /sftpForceCommand internal-sftp -u 2PermitTunnel noX11Forwarding no

Create SFTP directories

The frontends need an account with /sftp/<frontend-name>/uploads as home directory and a single accountis shared between probes ( uploads directory is needed as in chrooted environment home is not writeable). The laterneeds to access to all frontends, thus the associated home directory is simply /sftp/. For instance, a frontend namedfrontend, execute the following to create the directories:

$ sudo mkdir -pv /sftp/frontend/uploadsmkdir: created directory `/sftp'mkdir: created directory `/sftp/frontend'mkdir: created directory `/sftp/frontend/uploads'$ sudo chown -R root:sftpusers /sftp$ sudo chmod 0750 /sftp$ sudo chmod 0750 /sftp/frontend$ sudo chown -R frontend:sftpusers /sftp/frontend/uploads$ sudo chmod -R 0775 /sftp/frontend/uploads



Restart the service

You may want to restart the service:

$ sudo invoke-rc.d sshd restart

3.1.4 Installation

The Brain must be installed on a GNU/Linux distribution. With some efforts, it should be possible to run it on aMicrosoft Windows system, but this has not been tested yet.

This section describes how to get the source code of the application for the Brain and to install it.

Pre-requisites

We assume that you have a command line interface on your system with the following tools installed:

• python 3.4.x and newer

• python3-pip (see Install pip for the recommended way to install pip package manager)

Furthermore, we assume that you have created an user irma that will be used to run the python application.

Note: On GNU/Linux, one can create the user (and the group) irma with:

$ sudo adduser --system --no-create-home --group irma

Installation on GNU/Linux

On GNU/Linux system, we will assume that the code for the Brain will be installed in /opt/irma/irma-brain/current directory, which is the configured default installation directory.

The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp (-L):

# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/brain /opt/irma/irma-brain/current

We need few dependencies for future steps:

$ sudo apt-get install python3-virtualenv python3-dev

then, we need to install python dependencies manually in a virtualenv :

$ virtualenv --system-site-packages /opt/irma/irma-brain/current/venv$ /opt/irma/irma-brain/current/venv/bin/pip install -r /opt/irma/irma-brain/current/→˓requirements.txt[...]

If everything has gone well, you should have installed the python application on your system. The next step is to installthe other components it relies on and finally to configure it for your platform.

3.1. Brain 17

https://pip.pypa.io/en/latest/installing.html


3.1.5 Configuration

The configuration file is located at config/brain.ini in the installation directory. Update it with your specificinfo.

Note: Detailed meaning of each field in config/brain.ini:

Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (experimental)

prefix string irma-brain: prefix to append to rsyslog entriesdebug boolean False | enable Debug logsql_debug boolean False | enable SQL debug log

celery_options concurrency integer 0 number of concurrent workers (0means nb of cores)

soft_time_limit integer 300 (sec) time limit before task soft interrupttime_limit integer 1500 (sec) time limit before task is killed

broker_brain host string hostname for the RabbitMQ serverport integer 5672 port for the RabbitMQ servervhost string virtual host configured for brainusername string username used for brain on the Rab-

bitMQ serverpassword string password used for brain on the Rab-

bitMQ serverqueue string queue to poll new tasks on the Rab-

bitMQ serverbroker_probe host string hostname for the RabbitMQ server

port integer 5672 port for the RabbitMQ servervhost string virtual host configured for probesusername string username used for probes on the

RabbitMQ serverpassword string password used for probes on the

RabbitMQ serverqueue string queue to poll new tasks on the Rab-

bitMQ serverbroker_frontend host string hostname for the RabbitMQ server

port integer 5672 port for the RabbitMQ servervhost string virtual host configured for frontendusername string username used for frontend on the

RabbitMQ serverpassword string password used for frontend on the


bitMQ serversqldb dbms string sqlite dbapi engine

dialect string sqlalchemy dialectusername string database usernamepassword string database passwordhost string database hostdbname string /var/irma/

db/brain.dbdatabase name

tables_prefix string database tables prefixContinued on next page



Table 3.1 – continued from previous pageSection Key Type Default Descriptionftp protocol string “sftp” choose File Transfer Protocol

(“sftp” or “ftps”)ftp_brain host string hostname for the FTP server

port integer 21 port for the FTP serverauth string “password”| SFTP authentication method (“password”

or “key”)key_path string sftp private key absolute path

username string username used by probe on the FTP server

password string password used by the probe on theFTP server

interprocess_lock

path string /var/run/lock/irma-brain.lock

Concurrency file lock

ssl_config activate_ssl boolean False Enable RabbitMQ sslca_certs string RabbitMQ SSL certskeyfile string RabbitMQ SSL keyfilecertfile string RabbitMQ SSL certfile

Generate a SQLite database for scan tracking

You could easily generate the user database by running the following command. The path of the database is takenfrom the configuration file and the folder where the database is going to be stored must be created beforehand.

Note: The default path for the database is /var/irma/db/ make sure it exists before creating user database.

$ cd /opt/irma/irma-brain/current/$ ./venv/bin/python -m scripts.create_userusage: create_user <username> <rmqvhost> <ftpuser>

with <username> a string<rmqvhost> the rmqvhost used for the frontend<ftpuser> the ftpuser used by the frontend

example: create_user test1 mqfrontend frontend

To create an entry in the database for the frontend named frontend and which uses the mqfrontend virtual hoston the RabbitMQ server, simply run the following commands:

$ ./venv/bin/python -m scripts.create_user frontend mqfrontend frontend

Note: There is a limitation due to SQLite. The folder where the database is stored, plus the database file must bewritable by the user running the worker:

$ sudo chown irma:irma /var/irma/db/brain.db$ sudo chmod a+w /opt/irma/irma-brain

3.1. Brain 19


3.1.6 Installation Checks

Celery Workers

Before going further, you should check that the python applications manages to communicate with the RabbitMQserver through Celery. To ensure that, from the installation directory, execute both Celery workers:

On GNU/Linux:

$ cd /opt/irma/irma-brain/current$ ./venv/bin/python -m brain.scan_tasks

-------------- celery@brain v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2-- * - **** ---- ** ---------- [config]- ** ---------- .> app: scantasks:0x7fbd7ee4c350- ** ---------- .> transport: amqp://brain:**@127.0.0.1:5672/mqbrain- ** ---------- .> results: amqp://- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> brain exchange=celery(direct) key=brain

[2016-07-15 15:00:36,155: WARNING/MainProcess] celery@brain ready.

This worker is responsible for splitting the whole scan job in multiples job per probe per file.

$ cd /opt/irma/irma-brain/current$ ./venv/bin/python -m brain.results_tasks

-------------- celery@brain v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2-- * - **** ---- ** ---------- [config]- ** ---------- .> app: resultstasks:0x7fa68f9aa590- ** ---------- .> transport: amqp://probe:**@127.0.0.1:5672/mqprobe- ** ---------- .> results: disabled://- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> results exchange=celery(direct) key=results

[2016-07-15 14:59:01,799: WARNING/MainProcess] celery@brain ready.

And this worker is responsible for collecting and tracking results.

If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using.

SFTP accounts

Try to login as frontend and upload a sample file in home dir (should raise an error as it is non writeable) then inuploads dir.



$ sftp frontend@localhostfrontend@localhost's password:Connected to localhost.sftp> put testUploading test to /testremote open("/test"): Permission deniedsftp> lsuploadssftp> cd uploads/sftp> put testUploading test to /uploads/testtest→˓100% 10 0.0KB/s 00:00

3.1.7 Running Brain applications at startup

We have ensured that the freshly installed Brain is ready to be integrated to your IRMA platform. Now, we can go astep further and make it launch automatically all daemons when the system starts up so you will not need to relaunchthem manually every time.

We are using supervisor to manage our celery daemons.

Installing Supervisor

Install it with pip (python2):

$ sudo pip install supervisor

We will create two new applications called scan_app and result_app.

Configure Scan APP

Create a file called scan_app located at /etc/supervisor/conf.d with the following content:

[program:scan_app]

numprocs = 1stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/scan_app.logcommand = /opt/irma/irma-brain/current/venv/bin/python -m brain.scan_tasksuser = irmaautostart = Truedirectory = /opt/irma/irma-brain/currentstdout_logfile = /var/log/supervisor/scan_app.log

Configure Result APP

Create a file called result_app located at /etc/supervisor/conf.d with the following content:

[program:result_app]

numprocs = 1

3.1. Brain 21


stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/result_app.logcommand = /opt/irma/irma-brain/current/venv/bin/python -m brain.results_tasksuser = irmaautostart = Truedirectory = /opt/irma/irma-brain/currentstdout_logfile = /var/log/supervisor/result_app.log

Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:

[...][include]files = /etc/supervisor/conf.d/*

Restart supervisor:

$ sudo service supervisor restart

Restart applications if needed (should be done automatically):

$ sudo supervisorctl restart allscan_app: stoppedresult_app: stoppedscan_app: startedresult_app: started

3.2 Frontend

The Frontend handles scan submission to the Brain, stores the results of the scanned files. These results can bedisplayed through a web graphical user interface or via the command line interface.

3.2.1 Architecture

Let us recall first the inner architecture of the Frontend. It uses multiple technologies with each a specific purpose:

• A client through which an user submits a file and get the analysis results. There are two clients bundled in therepository: a web user interface and a command-line client.

• A python-based restful API, served by a nginx web server and a uWSGI application server. It gets the results ofa file scan by querying a database.

• A worker that will handle scan submission to the Brain and store the results of analyzes scheduled by the Brain.The worker relies on Celery, a python-based distributed task queue.

• An database server (PostgreSQL) is used to store results of analyzes made on each file submitted either by theweb graphical interface or the CLI client.

3.2.2 Installation

The Frontend must be installed on a GNU/Linux system. With some efforts, it should be possible to run it on aMicrosoft Windows system, but this has not been tested yet.

This section describes how to get the source code of the application and to install it.



Pre-requisites


• python 3.4.x and newer






On GNU/Linux system, we will assume that the code for the Frontend will be installed in /opt/irma/irma-frontend/current directory, which is the configured default installation directory.


$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/frontend /opt/irma/irma-frontend/current




$ virtualenv --system-site-packages /opt/irma/irma-frontend/current/venv$ /opt/irma/irma-frontend/current/venv/bin/pip install -r /opt/irma/irma-frontend/→˓current/requirements.txt[...]

Building the web client

The first step that should be performed when installing the frontend from the code source is to build the web clientwhich is composed of static HTML files mixed with javascript web framework (in particular, AngularJS).

One must install first some tools that are necessary to build the whole web client:

$ curl -sL https://deb.nodesource.com/setup | sudo bash -[...]$ sudo apt-get install -y nodejs[...]$ curl -sL https://www.npmjs.org/install.sh | sudo bash -[...]

Once the tools installed, you can build the static files for the web user interface with the following commands, executedfrom the installation directory:

3.2. Frontend 23



$ cd irma-frontend/current/web$ sudo npm install$ sudo node_modules/.bin/bower install --allow-root$ sudo node_modules/.bin/gulp dist

A new directory or an updated directory web/dist should appear now with HTML and javascript files that havebeen minified and “obfuscated” by gulp. For more details on the web interface, please refer to the dedicated chapter.

If everything has gone well, you should have installed the python application on your system. The next step is toconfigure it for your platform and to install the other components it relies on.

3.2.3 Configuration

The configuration file is located at config/frontend.ini in the installation directory.

Note: Detailed meaning of each field in config/frontend.ini:

Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (ex-

perimental)prefix string irma-frontend: prefix to append to

rsyslog entriesdebug boolean False enable Debug logsql_debug boolean False enable SQL debug

logsqldb username string database username

password string database passwordhost string database hostdbname string database nametables_prefix string database tables pre-

fixsamples_storage path string Samples storage

pathcelery_brain timeout integer 60 (sec) time before consid-

ering that the brainhas timed-out

celery_frontend timeout integer 30 (sec) time before consid-ering that the fron-tend has timed-out

celery_options concurrency integer 0 number of con-current workers (0means nb of cores)

soft_time_limit integer 300 (sec) time limit beforetask soft interrupt

time_limit integer 1500 (sec) time limit beforetask is killed

beat_schedule string /var/irma/frontend_beat_schedule

celery beat schedulefile

broker_brain host string hostname for theRabbitMQ server

port integer 5672 port for the Rab-bitMQ server

Continued on next page



Table 3.2 – continued from previous pageSection Key Type Default Description

vhost string virtual host config-ured for brain

username string username used forbrain on the Rab-bitMQ server

password string password used forbrain on the Rab-bitMQ server

queue string queue to poll newtasks on the Rab-bitMQ server

broker_frontend host string hostname for theRabbitMQ server

port integer 5672 port for the Rab-bitMQ server

vhost string virtual host config-ured for this fron-tend

username string username used forthis frontend on theRabbitMQ server

password string password used forthis frontend on theRabbitMQ server

queue string queue to poll newtasks on the Rab-bitMQ server

ftp protocol string “sftp” choose File TransferProtocol (“sftp” or“ftps”)

ftp_brain host string hostname for theFTP server

port integer 22 port for the FTPserver

auth string “password” SFTP authenti-cation method(“password” or“key”)

key_path string sftp private key ab-solute path

username string username used bythis frontend on theFTP server

password string password used bythis frontend on theFTP server

cron_clean_file_age clean_db_file

_max_age

integer 0 remove file after Xdays (0 means dis-abled)

Continued on next page

3.2. Frontend 25


Table 3.2 – continued from previous pageSection Key Type Default Description

clean_db_cron_hour

string 0 cron hour settings

clean_db_cron_minute

string 0 cron minute settings

clean_db_scan_day_of_week

string * cron day of weeksettings

cron_clean_file_size clean_fs_max

_size

string “0” space’smaxi-mumsize(bytes)dedi-cated tothe filesystemcronhoursettings

clean_fs_size_cron_hour

string *

clean_fs_size_cron_minute

string 0 cron minute settings

clean_fs_size_cron_day_of_week

string * cron day of weeksettings

interprocess_ lock path string /var/run/lock/ir ma-frontend.lock

Concurrency filelock

ssl_config activate_ssl boolean False Enable RabbitMQssl

ca_certs string RabbitMQ SSLcerts

keyfile string RabbitMQ SSL key-file

certfile string RabbitMQ SSLcertfile

Note: The default path for samples is /var/irma/samples/ make sure it exists with correct rights for irma user beforelaunching your first scan.

3.2.4 Installing and configuring uWSGI

The restful API is served by an uWSGI application server. This section will explain how to install an uWSGI serverand configure it for the Frontend.



Installation

On Debian, installing uWSGI to serve a python application is pretty straightforward:

$ cd /opt/irma/irma-frontend/current$ ./venv/bin/pip install https://projects.unbit.it/downloads/uwsgi-lts.tar.gz[...]

Please refer to the documentation of your preferred distribution to install an uWSGI server on it.

3.2.5 Installing SQL server

Note: Since version 1.5.0, IRMA required a postgreSQL server as we are using JSONB column.

The Frontend relies on a PostgreSQL database to keep track of all scans info. On Debian, one can install a PostgreSQLserver in few commands:

Install it with the following dependencies:

$ sudo apt-get install postgresql-9.4 python3-psycopg2[...]

You need to manually create the configured database and db user, the tables are automatically created on uwsgi startup.

3.2.6 Running Frontend applications at startup

We have ensured that the freshly installed Frontend is ready to be integrated to your IRMA platform. Now, we can goa step further and make it launch automatically all daemons when the system starts up so you will not need to relaunchthem manually every time.

We are using supervisor to manage our python daemons (uswgi for IRMA API and celery for IRMA frontend workers).

Installing Supervisor

Install it with apt:

$ sudo apt-get install python3-virtualenv python3-dev$ sudo pip install supervisor (python2 only)

We will create two new applications called frontend_api and frontend_app. Frontend_api is the python uwsgi serverand frontend_app the frontend celery daemon.

Configure Frontend API

Create a file called frontend_api located at /etc/supervisor/conf.d with the following content:

[program:frontend_api]

numprocs = 1redirect_stderr = Truestopwaitsecs = 600killasgroup = True

3.2. Frontend 27


stderr_logfile = /var/log/supervisor/frontend_api.logstopsignal = INTcommand = /opt/irma/irma-frontend/current/venv/bin/uwsgi -s 127.0.0.1:8081 --disable-→˓logging --master --workers 4 --need-app --chdir /opt/irma/irma-frontend/current --→˓home /opt/irma/irma-frontend/current/venv --python-path /opt/irma/irma-frontend/→˓current/venv --wsgi-file api/base.py --callable __hug_wsgi__ --lazy --offload-→˓threads 4user = irmaautostart = Truestdout_logfile = /var/log/supervisor/frontend_api.log

Configure Frontend APP

Create a file called frontend_app located at /etc/supervisor/conf.d with the following content:

[program:frontend_app]

numprocs = 1stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/frontend_app.logcommand = /opt/irma/irma-frontend/current/venv/bin/python -m api.tasks.frontend_appuser = irmaautostart = Truedirectory = /opt/irma/irma-frontend/currentstdout_logfile = /var/log/supervisor/frontend_app.log

Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:

[...][include]files = /etc/supervisor/conf.d/*

Restart supervisor:

$ sudo service supervisor restart

Restart applications if needed (should be done automatically):

$ sudo supervisorctl restart allfrontend_api: stoppedfrontend_app: stoppedfrontend_app: startedfrontend_api: started

3.2.7 Installing and configuring nginx

In the Frontend, we use a nginx web server to serve the uWSGI application and the static web site that query the APIin order to get results of scanned files and to present them to the user.

Installation

On Debian, installing nginx is done with few commands:



$ sudo apt-get install nginx[...]

Please refer to the documentation of your preferred distribution to install an uWSGI server on it.

Configuration

We provide several template scripts in the extras/ repository located at the root of the installation directory. Tem-plates for nginx are located in extras/nginx/. Copy the file to sites-available in nginx configurationfolder:

$ sudo cp extras/nginx/frontend /etc/nginx/sites-available/

The template is configured for a default installation of the frontend in /opt/irma/irma-frontend/current.You may need to edit it for you setup. In particular, please ensure that the root directive points to the folder web/dist (for a production ready version of the web site) or web/app (for a development version of the web site) inyour installation directory.

Activate the web site

By default, all websites in apps-available are not activated. One can enable the website described in frontendconfiguration file by creating a soft-link into the sites-enabled folder:

$ sudo ln -s /etc/nginx/sites-available/frontend /etc/nginx/sites-enabled/frontend

Note: The case of HTTPs

A template to set up a HTTPs server with nginx is also provided in the extras/nginx folder. Here is the way tosetup it:

$ sudo cp extras/nginx/frontend-https /etc/nginx/sites-available/$ sudo ln -s /etc/nginx/sites-available/frontend-https /etc/nginx/sites-enabled/→˓frontend-https$ sudo mkdir /etc/nginx/certificates/$ cd /etc/nginx/certificates/

Generate the required Diffie Hellman Ephemeral Parameters:

$ sudo openssl dhparam -out dhparam.pem 4096

Finally, generate a self signed certificate:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -subj "/CN=$(hostname --→˓fqdn)/" -keyout frontend.key -out frontend.crt

Relaunch the service

The final step is to relaunch the service:

$ sudo invoke-rc.d nginx restart

3.2. Frontend 29



Celery Workers

Before going further, you should check that the python application manages to communicate with the RabbitMQ serverand the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:

On GNU/Linux:

$ cd /opt/irma/irma-frontend/current$ ./venv/bin/python -m api.tasks.frontend_app

-------------- celery@frontend v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6-- * - **** ---- ** ---------- [config]- ** ---------- .> app: frontend.tasks:0x1e18750- ** ---------- .> transport: amqp://probe:**@brain:5672/mqfrontend- ** ---------- .> results: disabled- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> frontend exchange=celery(direct) key=frontend

[2014-08-20 15:28:58,745: WARNING/MainProcess] celery@frontend ready.

If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using. Let us note that the Celery worker gives us useful information onthe analyzer that are enabled.

SFTP accounts

Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.

$ sftp <user>@<hostname of the brain>

FTP-TLS accounts

Additionnally, if you have configured IRMA to use FTP-TLS, you can check whether the configured account is valid.On Debian, this can be done with the ftp-ssl package:

$ sudo apt-get install ftp-ssl[...]$ ftp-ssl <hostname of the brain>Connected to brain.220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------220-You are user number 1 of 50 allowed.220-Local time is now 18:55. Server port: 21.220-This is a private system - No anonymous login220-IPv6 connections are also welcome on this server.220 You will be disconnected after 15 minutes of inactivity.Name (brain:root): frontend-ftp500 This security scheme is not implemented234 AUTH TLS OK.



[SSL Cipher DHE-RSA-AES256-GCM-SHA384]200 PBSZ=0200 Data protection level set to "private"331 User probe OK. Password requiredPassword: frontend-ftp-password230 OK. Current directory is /Remote system type is UNIX.Using binary mode to transfer files.ftp>

Restful API

One can verify that the restful API is up and running by querying a specific route on the web server or by checking thesystem logs:

$ curl http://localhost/api/v1.1/probes{"total": 9, "data": ["ClamAV", "ComodoCAVL", "EsetNod32", "FProt", "Kaspersky",→˓"McAfeeVSCL", "NSRL", "StaticAnalyzer", "VirusTotal"]}

$ sudo cat /var/log/supervisor/frontend_api.log[...]added /opt/irma/irma-frontend/current/venv/ to pythonpath.

*** uWSGI is running in multiple interpreter mode ***spawned uWSGI master process (pid: 3943)spawned uWSGI worker 1 (pid: 3944, cores: 1)spawned uWSGI worker 2 (pid: 3945, cores: 1)spawned uWSGI worker 3 (pid: 3946, cores: 1)spawned uWSGI worker 4 (pid: 3947, cores: 1)mounting frontend/api/base.py on /apimounting frontend/api/base.py on /apimounting frontend/api/base.py on /apimounting frontend/api/base.py on /apiWSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3945→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3946→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3944→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3947→˓(default app)

3.3 Probe

The Probes are python-based application that host a single or multiple analyzers. Each analyzer listens on a specificwork queue and waits for an analysis to be scheduled by the Brain through Celery, an open source task framework forPython. Python version should be at least 3.4 on linux, 3.5 on windows.

3.3.1 Architecture

Probes are mainly Celery workers that handle scan requests from Brain

3.3. Probe 31


3.3.2 Installation

Probes can be installed either on GNU/Linux and on Microsoft Windows systems. The installation procedure maydiffers between the two systems.

Downloading the source code from github.com

The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp1 (-L):

# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current


• python 3.4.x and newer on Linux

• python 3.5.x and newer on Windows (see Python for Windows for prebuild MSI installer)





Installation on Microsoft Windows

On windows system, we will assume that the code for the Probe will be installed at the root of the C:\ drive, namelyin C:\irma\irma-probe\current.

$ C:\Python27\Scripts\pip.exe install virtualenv$ C:\Python27\Scripts\virtualenv C:\irma\irma-probe\current\venv

$ cd C:\irma\irma-probe\current\$ .\venv\Scripts\pip.exe install -r requirements.txt[...]


On GNU/Linux system, we will assume that the code for the Probe will be installed in /opt/irma/irma-probe/current directory.


1 On Microsoft Windows, a Linux-like lightweight command line interface can be installed by installing MSYS or Git for windows.


https://www.python.org/downloads/windows/


http://www.mingw.org/wiki/MSYS


# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current




$ virtualenv --system-site-packages --python=/usr/bin/python3 /opt/irma/irma-probe/→˓current/venv$ /opt/irma/irma-probe/current/venv/bin/pip install -r /opt/irma/irma-probe/current/→˓requirements.txt[...]

If everything has gone well, the python application is now installed on your system. The next step is to configure it foryour platform and to enable the analyzers you need.

3.3.3 Configuration

The configuration file is config/probe.ini located in the installation directory.

Note: We recall in the following the meaning of each field in config/probe.ini:

Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (experimental)

prefix string irma-probe: prefix to append to rsyslog entriescelery_options concurrency integer 0 number of concurrent workers (0

means nb of cores)soft_time_limit integer 300 (sec) time limit before task soft interrupttime_limit integer 1500 (sec) time limit before task is killed

broker probe host string hostname for the RabbitMQ serverport integer 5672 port for the RabbitMQ servervhost string virtual host configured for probesusername string username used for probes on the

RabbitMQ serverpassword string password used for probes on the


bitMQ serverftp_brain host string hostname for the FTP server

port integer 21 port for the FTP serverauth string “password”| SFTP authentication method (“password”

or “key”)key_path string sftp private key absolute path

username string username used by probe on the FTPserver

password string password used by the probe on theFTP server

3.3. Probe 33


3.3.4 Enabling Analyzers

The python application auto-discovers available analyzers. As long as the requirements are fulfilled for an analyzer,it is automatically discovered and enabled. By default, no analyzers are available. The following enumerates therequirements to enable each analyzer bundled with the application.

ClamAV - GNU/Linux

To enable ClamAV on Debian Stable distribution, one should install several packages:

$ sudo apt-get install clamav-daemon[...]$ sudo freshclam[...]$ sudo service clamav-daemon restart[...]

ComodoCAVL - GNU/Linux

Comodo Antivirus for Linux can be downloaded from the Comodo’s download page. The following instruction enableto install the Debian package. By default, the binaries are installed in /opt/COMODO/ directory. As ComodoCAVLis not packaged for the current Debian Stable distribution, we must install it manually:

$ sudo apt-get install binutils$ ar x cav-linux_1.1.268025-1_iXXX.deb$ sudo tar xvf ~/data.tar.gz -C /[...]

Updates for the database can be performed with the following command:

$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/COMODO/menu/comodo-updater[...]

Note: Dependencies to update the database

To be able to update the database using the updater provided with Comodo Antivirus for Linux, some dependeciesneeded for the graphical interface may be missing from the distribution. On Debian Stable system, one can installthem with:

$ sudo apt-get install libqt4-sql libqt4-network libqtgui4

EsetNod32 - GNU/Linux

One can request a trial version of Eset Nod32 Business Edition for Linux on the Eset download page. Once down-loaded, the anti-virus can be installed with the following commands on Debian. Just follow the typical installation onthe GUI:

$ sudo chmod u+x eset_nod32av_64bit_en.linux$ sudo apt-get install libgtk2.0-0 libc6-i386$ sudo ./eset_nod32av_64bit_en.linux[...]


http://www.comodo.com/home/internet-security/antivirus-for-linux.php

http://www.eset.com/int/download/home/detail/family/71/


Binaries should be installed in /opt/eset/esets directory. Updates are performed from the GUI:

$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/eset/esets/bin/esets_gui

Note: Disabling the antivirus daemon

To avoid the anti-virus to protect your system at startup, we deliberately disabled the script used to launch the anti-virusearly at boot:

$ sudo service esets stop$ sudo mv /etc/init.d/esets /etc/init.d/esets.disable

F-Prot - GNU/Linux

A copy of F-PROT anti-virus for Linux workstations is available on the F-PROT download page.

The binaries should be installed in /usr/local/f-prot to make the python application detect it automatically.

$ sudo tar xvf fp-Linux.x86.32-ws.tar.gz -C /usr/local/

To launch an update, a configuration step is mandatory:

$ sudo cp /usr/local/f-prot/f-prot.conf.default /etc/f-prot.conf

An update is launched with:

$ sudo ./fpupdateERROR: ld.so: object 'libesets_pac.so' from /etc/ld.so.preload cannot be preloaded:→˓ignored.[...]

Note: Error

If you see an error message like:

DownloadingWarning: Network - Connection failed (18), trying again...Downloading updateError: Update - Bad mergefile

Just relaunch the script.

Note: Dependencies to update the database

To be able to update the database using the updater provided with Comodo install them with:

$ sudo apt-get install libc6-i386

McAfeeVSCL - GNU/Linux or Microsoft Windows

A free evaluation of McAfee VirusScan Command Line can be downloaded from the editor download page.

3.3. Probe 35

http://www.f-prot.com/download/home_user/download_fplinux.html

http://www.mcafee.com/apps/downloads/free-evaluations/


The binaries should be installed in /usr/local/uvscan/ on GNU/Linux system and must be installed inC:\VSCL on Windows Systems. Let us note that updates must be performed manually. Anti-virus databases andengines can be downloaded here.

After downloading McAfee Virus Scan archive, create /usr/local/uvscan and extract the archive in it:

$ sudo mkdir /usr/local/uvscan$ sudo tar xvf vscl-XXX.tar.gz -C /usr/local/uvscan # replace using your values$ sudo chmod +x /usr/local/uvscan/uvscan

Extract also, using unzip program, the database:

$ sudo unzip avvepo7536dat.zip -d /usr/local/uvscan$ cd /usr/local/uvscan$ sudo unzip avvdat-XXXX.zip

Sophos - GNU/Linux or Microsoft Windows

A free evaluation of Sophos Endpoint Antivirus can be downloaded from the editor download page. You should receiveby email a username and a password to authenticate for updates.

It should be detected automatically by IRMA if the anti-virus is installed in its default installation directory.

On GNU/Linux:

• Download the archive for Linux, then execute:

$ tar zxf sav-linux-9-i386.tgz$ ./sophos-av/install.sh /opt/sophos-av --acceptlicence --autostart=False --→˓enableOnBoot=False --automatic --ignore-existing-installation$ /opt/sophos-av/bin/savconfig set EnableOnStart false$ /opt/sophos-av/bin/savconfig set PrimaryUpdateSourcePath "sophos:"$ /opt/sophos-av/bin/savconfig set PrimaryUpdateUsername "<your_username_from_email>"$ /opt/sophos-av/bin/savconfig set PrimaryUpdatePassword "<your_password_from_email>"$ /opt/sophos-av/bin/savupdate

Kaspersky - Microsoft Windows

A free evaluation of Kaspersky Internet Security can be requested on the editor download page. It should be detectedautomatically if the anti-virus is installed in its default installation directory.

Symantec - Microsoft Windows

The procedure to install a trial version of Symantec Endpoint Protection is particularly tedious. We will not documentits installation.

G Data - Microsoft Windows

A trial version of G Data is available on the editor download page<https://www.gdata.de/kundenservice/downloads.html>. It should be detected automatically if the anti-virus isinstalled in its default installation directory.


http://www.mcafee.com/apps/downloads/security-updates/security-updates.aspx

http://www.sophos.com/en-us/products/endpoint-antivirus/free-trial.aspx

http://usa.kaspersky.com/downloads/free-home-trials/Internet-security


VirusTotal - GNU/Linux or Microsoft Windows

The VirusTotal analyzer can be installed easily by downloading the python packages it depends on and by modifyingits configuration file. From the installation directory, one can execute:

On GNU/Linux:

$ pip install -r modules/external/virustotal/requirements.txt[...]

then update configuration file located at modules/external/virustotal/config.ini.

Note: Meaning of the fields in the configuration file

Section Option Type Default DescriptionVirusTotal apikey string api_key used to query VirusTotal APIVirusTotal private boolean use private api (need a private api key)

NSRL - GNU/Linux

The National Software Reference Library can be downloaded on the NIST’s web page. The provided files are storedin the RDS (Reference Data Set) format. To use this analyzer, one must build first the database. We use LevelDB asfast key-value storage library.

To build the dabatase, one must install first the dependencies:

$ pip install -r modules/database/nsrl/requirements.txt

A (not optimized and very slow) helper script is provided to build the database:

$ mkdir /home/irma/leveldb$ python -m modules.database.nsrl.nsrl create -t os NSRLOS.txt /home/irma/leveldb/os_→˓db$ python -m modules.database.nsrl.nsrl create -t manufacturer NSRLMfg.txt /home/irma/→˓leveldb/mfg_db$ python -m modules.database.nsrl.nsrl create -t product NSRLProd.txt /home/irma/→˓leveldb/prod_db$ python -m modules.database.nsrl.nsrl create -t file NSRLFile.txt /home/irma/leveldb/→˓file_db

Finally, one must indicate to the analyzer where to find the files for the database by filling the configuration file locatedat modules/database/nsrl/config.ini.

Note: Meaning of the fields in the configuration file

Section Option Type Default DescriptionNSRL nsrl_os_db string location of the OS database

nsrl_mfg_db string location of the Manufacturer databasensrl_file_db string location of the File databasensrl_prod_db string location of the Product database

3.3. Probe 37

http://www.nsrl.nist.gov/


Note: Error

If you see an error message like:

fatal error: Python.h: No such file or directory

Then you’ll need to install python3-dev package (for Debian like systems).

Note: leveldb.LevelDBError: IO error: /home/irma/leveldb/file_db/LOCK: Permission denied

If you encounter this problem, you likely have a problem with unix permissions. Please ensure that the folder is ownedby the user running the probes. On supervisord-based installation (default for vagrant/ansible scripts), the folderowner should be set to nobody. For init.d-based installation, it should be irma instead.

StaticAnalyzer - GNU/Linux or Microsoft Windows

The PE File analyzer adapted from Cuckoo Sandbox can be installed easily. One need to install the python packagesit depends on. From the installation directory, one can execute:

On GNU/Linux:

$ pip install -r modules/metadata/pe_analyzer/requirements.txt[...]

On Microsoft Windows, you need cygwin to successfully install the requirements (see python3-magic documentationfor installation details):

$ C:\Python27\Scripts\pip.exe install -r modules/metadata/pe_analyzer/requirements.txt[...]

Yara - GNU/Linux or Microsoft Windows

Please refer to modules/metadata/yara/README.md file for the documentation.

Guide on Debian (credits to Carbonn)

1. Installing dependencies

$ sudo apt-get install libtool automake bison

2. Installing Yara python modules

$ git clone https://github.com/plusvic/yara.git$ autoreconf -i --force$ ./configure$ make$ sudo make install$ python setup.py build$ sudo python setup.py install$ sudo ldconfig


https://github.com/ahupp/python-magic#dependencies


3. Configuring for IRMA

$ mkdir /opt/irma/yara_rules/$ cat /opt/irma/yara_rules/yara_rules.yar << EOF# Insert rule below inside the file

rule silent_banker : banker{

meta:description = "This is just an example"thread_level = 3in_the_wild = true

strings:$a = {6A 40 68 00 30 00 00 6A 14 8D 91}$b = {8D 4D B0 2B C1 83 C0 27 99 6A 4E 59 F7 F9}$c = "UVODFRYSIHLNWPEJXQZAKCBGMT"

condition:$a or $b or $c

}EOF


Celery Workers

Before going further, you should check that the python application manages to communicate with the RabbitMQ serverand the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:

On GNU/Linux:

$ cd /opt/irma/irma-probe/current$ ./venv/bin/python -m probe.tasksWARNING:root: *** [plugin] Plugin failed to load: Kaspersky miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: Sophos miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: Symantec miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: NSRL miss dependencies: leveldict→˓(ModuleDependency). See requirements.txt for needed dependenciesWARNING:root: *** [plugin] Plugin failed to load: VirusTotal miss dependencies: virus_→˓total_apis (ModuleDependency). See requirements.txt for needed dependenciesWARNING:root: *** [plugin] Plugin failed to load: StaticAnalyzer miss dependencies:→˓pefile (ModuleDependency). See requirements.txt for needed dependencies

-------------- celery@irma-probe v3.1.13 (Cipater)---- **** -------- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6-- * - **** ---- ** ---------- [config]- ** ---------- .> app: probe.tasks:0x1e18750- ** ---------- .> transport: amqp://probe:**@brain:5672/mqprobe- ** ---------- .> results: redis://brain:6379/1- *** --- * --- .> concurrency: 1 (prefork)-- ******* ----

3.3. Probe 39


--- ***** ----- [queues]-------------- .> ClamAV exchange=celery(direct) key=ClamAV

.> ComodoCAVL exchange=celery(direct) key=ComodoCAVL

.> EsetNod32 exchange=celery(direct) key=EsetNod32

.> FProt exchange=celery(direct) key=FProt

.> McAfeeVSCL exchange=celery(direct) key=McAfeeVSCL

[2014-08-19 18:25:06,469: WARNING/MainProcess] celery@irma-probe ready.

The equivalent command on Microsoft Windows is:

C:\> cd "C:\IRMA\irma-probe\current"C:\> .\venv\Scripts\python.exe -m probe.tasks[...]

If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using. Let us note that the Celery worker gives us useful information onthe analyzer that are enabled. As each analyzer connects to a dedicated queue, we can deduce, in this example, thatthe analyzers ClamAV, ComodoCAVL, EsetNod32, FProt and McAfeeVSCL are enabled and ready to serve.

SFTP accounts

Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.

$ sftp <user>@<hostname of the brain>

FTP-SSL accounts

Additionnally, if you have configured IRMA to use FTP-ssl, you can check whether the configured account is valid.On Debian, this can be done with the ftp-ssl package:

$ sudo apt-get install ftp-ssl[...]$ ftp-ssl <hostname of the brain>Connected to brain.220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------220-You are user number 1 of 50 allowed.220-Local time is now 18:55. Server port: 21.220-This is a private system - No anonymous login220-IPv6 connections are also welcome on this server.220 You will be disconnected after 15 minutes of inactivity.Name (brain:root): probe-ftp500 This security scheme is not implemented234 AUTH TLS OK.[SSL Cipher DHE-RSA-AES256-GCM-SHA384]200 PBSZ=0200 Data protection level set to "private"331 User probe OK. Password requiredPassword: probe-ftp-password230 OK. Current directory is /Remote system type is UNIX.Using binary mode to transfer files.ftp>


CHAPTER 4

Database migration

IRMA uses Alembic to manage and perform databases migration.

Note: Alembic is a useful tool to manage migration, but can’t surpass local engine implementation of SQL. AsSQLite doesn’t manage schema modifications such as ALTER_COLUMN, the whole migration system of IRMAwon’t support it. The preferred database engine is PostgreSQL.

You can still use SQLite, but you will be on your own for migrations.

Warning: Please note that most of the manipulations on this can and sometimes will alter your data. If you arenot sure about what you are doing, and even if you are sure, make backup.

4.1 Requirements

• Alembic

4.2 Content

Database migrations are managed in the frontend and brain IRMA components.

The files/directories used are:

alembic.iniextras/migration/

+- env.py+- script.py.mako+- versions/

+- <revision_1>.py

41

https://alembic.readthedocs.org/en/latest/

https://pypi.python.org/pypi/alembic


+- <revision_2>.py+- ...

Note: All the commands below will assert to be executed on top of this file system, as Alembic needs the alembic.ini configuration file.

You could also use the -c <path_to_conf_file>.

4.3 Usage

Alembic manage a ‘revision’ for each database evolution. These revisions are used to upgrade or downgrade thedatabase schema.

The command:

$ alembic current

. . . shows the current revision of the database.

The command to get the history of the latest alembic migrations is:

$ alembic history --verbose

4.3.1 Create database from scratch with Alembic

Configuration and creating database

Alembic will use the information in the [sqldb] section of the configuration files (respectively config/frontend.ini or conf/brain.ini for the repositories of the frontend or the brain components). Make surethey are accurate.

The database must already exist. This step is quite simple, the SQL command usually being:

sql$ CREATE DATABASE <db_name>;

Update your schema with Alembic

If you use a virtualenv, activate it. Then enter:

$ alembic upgrade head

Alembic applies each revision one after the other. At the end of the process, if no error occurs, your database shouldbe updated.

Note: You can update the database one revision at a time, or up to a specific revision. See the revisions section forfurther information.

42 Chapter 4. Database migration


4.3.2 If you already have a database WITHOUT Alembic

Alembic stores its current revision number in database. If your database doesn’t have this information, you are verylikely to encounter errors when using Alembic, as it will try to create already existing tables.

The easiest solution is to destroy your database and go for a fresh install.

Although, if you don’t want to lose your data, you could update the Alembic information manually.

You will need to:

1. Get the exact current Alembic revision of your database. Each migration file has a Revision ID in its header.Investigate the successive revisions to know which one matches your current database state.

2. Once you known your Alembic revision, run:

$ alembic stamp <your_alembic_revision_number>

3. Your database is now synchronized with Alembic! You should be able to use Alembic to upgrade/downgradeyour database now. Be aware that if the revision number you provided is false, you could encounter massiveerrors while attempting to upgrade/downgrade your database.

4.3.3 Generating a new revision

Creating a new revision can be done with the command:

$ alembic revision -m <revision_message>

This command produces a new <hash>_<revision_message>.py file in the extras/migration/versions/ directory. This file contains two functions upgrade and downgrade, respectively used to upgradethe database to the revision, or downgrade from it. These two functions are empty and must be completed with thedesired modifications (see the alembic documentation).

A revision could be produced automatically, from database metadata defined in the IRMA SQL objects descriptionthrough sqlalchemy, with the command:

$ alembic revision --autogenerate -m <revision_message>

These SQL objects are defined in:

• frontend/models/sqlobjects.py for the frontend,

• brain/models/sqlobjects.py for the brain.

Alembic scripts in IRMA repositories are already configured to use metadata defined in these files. You should be ableto use the --autogenerate option without further modifications.

Note: IRMA configuration allows to prefix table names through configuration. Our revision files use the function<frontend_or_brain>/config/parser.py:prefix_table_name to generate table names rather thankeeping alembic-generated plain string names. A good practice would be to keep using this function in revision files.

Warning: Alembic easily detects changes such as adding/removing columns, but could be blind on thin, innermodifications. Re-reading the auto-generated script is a strongly recommended step before actually performingthe migration.

See the alembic documentation for more information.

4.3. Usage 43

https://alembic.readthedocs.org/en/latest/ops.html

https://alembic.readthedocs.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect


Warning: Database modifications using ALTER_COLUMN (such as changing the type of a column) can’t beperformed on SQLite databases. Be aware of this limitation if you absolutely want

to use migration scripts with this SQL engine.

4.3.4 Migrating between revisions

Once the revision is properly described, the migration is performed with:

$ alembic upgrade head

Alembic allows to migrate the database to any revision, relatively to the current revision or absolutely. Several exam-ples:

$ alembic upgrade +4$ alembic downgrade base$ alembic upgrade <revision_number>+3

4.4 Tips and tricks

Note: Don’t trust Alembic too much. It is nothing more than a tool, without any comprehension on the code.Cautiously read the revision scripts it generates.

Note: Database migration is hardly ever a painless step. Be sure to:

1. save your data before performing a migration,

2. test your application after the migration to ensure its compatibility with the new data schemes.

Note: With a PostgreSQL database, the Float type is tolerated but the real type name used by the databaseis Real. It means that SQL objects described in sqlalchemy with Float columns will be properly applied indatabase, but at each autogenerate revision, alembic will see Real type in database, against Float type in thecode metadata, and so will perform each time a useless alter_column from Real to Float. This problem couldbe avoided (with PostgreSQL) by declaring Real instead of Float.

See this page for more information on PostgreSQL numeric types.

Note:

Alembic can’t directly deal with many somehow complex operations, such as type migration with notrivial cast. In these cases, the operation must be manually described with a raw SQL command (whichcould be database-dependent).

For instance, alembic can’t perform the migration from real to datetime:

> alembic.alter_column('table', 'column',existing_type=sqlalchemy.REAL(),


http://www.postgresql.org/docs/9.1/static/datatype-numeric.html


type_=sqlalchemy.DateTime(),existing_nullable=False)

. . . because of an error a column "column" cannot be cast automatically to typetimestamp with time zone.

A proper migration for PostgreSQL would be (in Python):

> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE TIMESTAMP WITHOUT→˓TIME ZONE USING to_timestamp(column)')

And the reverse code to downgrade the migration could be:

> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE REAL USING→˓extract(epoch from column)')

Note: Rather than managing migrations directly with Alembic, we could generate SQL migration revision to be useddirectly on database with the command:

$ alembic upgrade <revision> --sql > migration.sql

Note: Deleting a revision R is simple:

• downgrade the database to the revision before R-1 the revision you want to delete;

• if any, edit the script of the following revision R+1 and update the down_revision variable to match therevision number of revision R-1;

• delete the script of the revision R you want to delete;

• upgrade your database.

The deleted revision want be applied any more.

4.4. Tips and tricks 45

CHAPTER 5

To evolve IRMA

5.1 Adding a new probe

5.1.1 Writing a Plugin for the probe

Note: To be a valid probe module, IRMA expects it to have a predefined structure. To save time, one can get aminimal working structure from the skeleton plugin. The new plugin is stored in the appropriate sub-directory of thedirectory probe/modules according to the type of the new probe (antivirus, metadata, external. . . ).

For a probe that is not a antivirus

1. Copy the directory skeleton to the new module (appropriate localisation). Example with a module my_module withmetadata type :

$cp -r probe/modules/custom/skeleton/ probe/modules/metadata/my_module

2. If there are packages to install, specify them in the file requirements.txt. Otherwise remove the file

3. Adjust the file plugin.py according to the module :

• Adjust the class’s name with the name of your probe

• Fill in the fields of the class :- _plugin_name_ = [the plugin name] - _plugin_display_name_ = [the field_name of the class of the probe] - _plugin_version_ = [the version number] - _plugin_category = [the typeof the probe: IrmaProbeType.] - _plugin_description = [uick description] - _plugin_dependencies = [listof dependencies: platform, binary or/and file]

=> if used import from lib.plugins PlatformDependency, BinaryDependency or/and FileDepen-dency

– _mimetype_regexp = [mimetype corresponding]

4. Implement the functions corresponding to the type of the plugin

47


For an antivirus

In the case of an antivirus, it is a little different because an Antivirus class was created to avoid code’s duplication.You can use the skeleton below:

plugin.py:

## Copyright (c) 2013-2018 Quarkslab.# This file is part of IRMA project.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License in the top-level directory# of this distribution and at:## http://www.apache.org/licenses/LICENSE-2.0## No part of the project, including this file, may be copied,# modified, propagated, or distributed except according to the# terms contained in the LICENSE file.

from .skeleton import Skeleton

from lib.plugins import PluginBase, PluginLoadErrorfrom lib.irma.common.utils import IrmaProbeType

class SkeletonPlugin(PluginBase, Skeleton):

# =================# plugin metadata# =================_plugin_name_ = "Skeleton"_plugin_display_name_ = Skeleton._name_plugin_author_ = "IRMA (c) Quarkslab"_plugin_version_ = "1.0.0"_plugin_category_ = "custom"_plugin_description_ = "Plugin skeleton"_plugin_dependencies_ = []_mimetype_regexp = None

# =============# constructor# =============

def __init__(self):self.module = Skeleton()

skeleton.py:

## Copyright (c) 2013-2018 Quarkslab.# This file is part of IRMA project.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License in the top-level directory# of this distribution and at:#

48 Chapter 5. To evolve IRMA


# http://www.apache.org/licenses/LICENSE-2.0## No part of the project, including this file, may be copied,# modified, propagated, or distributed except according to the# terms contained in the LICENSE file.

import logging

# Choose the class you need to inherit fromfrom modules.antivirus.base import AntivirusUnix, AntivirusWindows

log = logging.getLogger(__name__)

# Inhererit from AntivirusUnix or AntivirusWindows according to your plateformclass Skeleton(Antivirus):

name = "Skeleton for Antivirus"

# ==================================# Constructor and destructor stuff# ==================================

def __init__(self, *args, **kwargs):# class super class constructorsuper().__init__(*args, **kwargs)

# do your initialization stuff

# Finally initialize all the attributesself._init_attributes()

The recipe is the same, the files with the corresponding module name and differents fields need to be updated. Theattributes in Antivirus._attributes are meant to be defined by the instanciation. One can either:

• leave it blank, in this case the super class will assign it a default value (eg. "unavailable" for self.version);

• define it directly (eg. self.scan_path = Path("/opt/skeleton/skeleton"));

• define a function to be called to assign it (eg. def get_scan_path(self): ...), the super class willtake care of calling it and handling exceptions.

5.1.2 Testing the new plugin

Before testing, module’s necessary stuff (binaries, files, etc) must be provisioned to the VM.

$cd ansible$vagrant rsync$vagrant ssh$sudo su deploy$cd /opt/irma/irma-probe/current$venv/bin/python -m extras.tools.run_module

This last command lists available modules.

Now, if the new module is available, its launching can be done:

$venv/bin/python -m extras.tools.run_module my_module file

5.1. Adding a new probe 49


5.1.3 Automatic provisioning

Creating a new role

Create a new directory with this structure:

cd ansibletree roles/quarsklab.my_moduleroles/quarkslab.my_module/+-- defaults| +-- main.yml+-- tasks

+-- main.yml

tasks/main.yml is the default entry point for a role containing Ansible tasks. In this file, write the instruction to installthe module. Add the file tasks/update.yml to write the informations for the update if necessary. In defaults/main.yml itis usual to store default variables for this role. If there are particular instructions, for example how to obtain a licencefor a antivirus, add a README file.

Invoking the module role

Modify playbooks/provisioning.yml : add the module

-name : my_modulehosts: my_moduleroles:- { role: quarkslab.my_module, tags: 'my_module'}

If a task update was defined, add the module in playbooks/updating.yml :

-name : my_modulehosts: my_moduleroles:- { role: quarkslab.module, tags: 'my_module', task_from : update}

Defining hosts

Modify the environment to add the new probe.

For example for the allinone_dev :

$ cat environments/allinone_dev.yml[ ... snip ... ]

virustotal:- brain.irma

my_module:- brain.irma

"probe:children":- clamav- comodo- mcafee- static-analyzer- virustotal- my_module

50 Chapter 5. To evolve IRMA

CHAPTER 6

References

6.1 Disclaimer

IRMA is distributed as it is, in the hope that it will be useful, but without any warranty neither the implied mer-chantability or fitness for a particular purpose.

Whatever you do with this tool is uniquely your own responsibility.

6.2 License

IRMA source code is licensed under Apache License, version 2.0.

The full license text can be found below (Apache License, version 2.0).

6.3 Apache License, version 2.0

6.4 Authors

51


52 Chapter 6. References

CHAPTER 7

Frequently Asked Questions

7.1 Playing with tags

Tags are available in IRMA from version 1.3.0

7.1.1 Creating a tag

You could create tags by using the command line tools

>>> from irmacl.helpers import *>>> tag_list()[]

>>> tag_new("archive"){u'text': u'archive', u'id': 1}

>>> tag_list()[Tag archive [1]]

or directly from your terminal by using curl and posting a json with ‘text’ key:

$ curl -H "Content-Type: application/json; charset=UTF-8" -X POST -d '{"text":"<your→˓tag>"}' http://172.16.1.30/api/v1.1/tags

Note: There is currently no way to create a tag directly from the web IHM.

7.1.2 Tagging a File

Directly in web IHM, once you are on a file details page:

53

https://github.com/quarkslab/irma-cli


Just click the tag bar and you will see all available tags. You could add multiple tags.

It is also possible to add a tag through command line tools:

54 Chapter 7. Frequently Asked Questions


>>> from irmacl.helpers import *>>> file_tag_add?Signature: file_tag_add(sha256, tagid, verbose=False)Docstring:Add a tag to a File

:param sha256: file sha256 hash:type sha256: str of (64 chars):param tagid: tag id:type tagid: int:return: No return

>>> file_tag_add("346ae869f7c7ac7394196de44ab4cfcde0d1345048457d03106c1a0481fba853",1)

7.1.3 Searching by tag

You could specify one or more tags while searching for files too:

choose your tag list then hit the search button:

or by command line:

>>> from irmacl.helpers import *>>> file_search(tags=[1])(1, [<irma.apiclient.IrmaResults at 0x7f079ca23890>])

7.1. Playing with tags 55


7.2 SSL settings

7.2.1 Making RabbitMQ running SSL

Certificates generation

See rabbitmq detailled guide on how to generate server and clients certificates here

Update server settings on brain

Copy the RabbitMQ configuration template provided in “./extras/rabbitmq” to “/etc/rabbitmq”

Restart RabbitMQ:

$ sudo service rabbitmq-server restart

Allow irma to use SSL with RabbitMQ

Note: Frontend configuration file is <irma dir>/config/frontend.ini

Brain configuration file is <irma dir>/config/brain.ini

Probe configuration file is <irma dir>/config/probe.ini

Update PORT in configuration file from 5672 to 5671. Except if you change the default port defined in the theRabbitMQ configuration template provided in the precedent paragraph.

[broker_xxxx]host = 127.0.0.1port = 5671

Update activate_ssl switch in configuration file:

[ssl_config]activate_ssl = yesca_certs =keyfile =certfile =

Put the SSL certificates (ca_cert, key_file, cert_file) in <irma dir>/ssl Update ca_certs, keyfile andcertfile in configuration file according to the filenames in “./ssl”

[ssl_config]activate_ssl = yesca_certs = <ca_cert_filename>keyfile = <key_filename>certfile = <cert_filename>

Note: If you are switching to ssl from an already running no_ssl version, please do the following on irma-brainRabbitMQ server:


https://www.rabbitmq.com/ssl.html


$ sudo rabbitmqctl stop_app$ sudo rabbitmqctl reset$ sudo rabbitmqctl start_app# create again the RabbitMQ vhosts, usernames and passwords:$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe probe mqprobe$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain brain mqbrain$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh frontend frontend mqfrontend

7.3 How to debug

7.3.1 Switch debug log on

Configuration file for frontend, brain and probe is located by default in the config folder and is named respectivelyfrontend.ini, brain.ini and probe.ini.

To turn on debug log just add the following line:

[log]syslog = 0debug = 1

and restart all related applications.

To turn on SQL debug log (warning: its verbose) just add the following line:

[log]syslog = 0debug = 1sql_debug = 1

and restart all related applications.

7.3.2 Debug a probe

Open a session on the probe machine and change directory to the irma-probe location. Try the run_module tool on afile to see what analyzer is detected and what is its output on a file.

$ sudo su deploy$ cd /opt/irma/irma-probe/current$ ./venv/bin/python -m extras.tools.run_module

[...]usage: run_module.py [-h] [-v]

{Unarchive,StaticAnalyzer,ClamAV,VirusTotal} filename[filename ...]

run_module.py: error: too few arguments

Here 4 probes are automatically detected. Now try one on a file:

$ ./venv/bin/python -m extras.tools.run_module ClamAV requirements.txt{'database': {'/var/lib/clamav/bytecode.cvd': {'ctime': 1458640823.285298,

'mtime': 1458640823.069295,'sha256':

→˓'82972e6cc5f1204829dba913cb1a0b5f8152eb73d3407f6b86cf388626cff1a1'},

7.3. How to debug 57


'/var/lib/clamav/daily.cvd': {'ctime': 1458640822.8932924,'mtime': 1458640822.6692889,'sha256':

→˓'9804c9b9aaf983f85b4f13a7053f98eb7cca5a5a88d3897d49b22182b228885f'},'/var/lib/clamav/main.cvd': {'ctime': 1458640821.6972747,

'mtime': 1458640813.9771628,'sha256':

→˓'4a8dfbc4c44704186ad29b5a3f8bdb6674b679cecdf83b156dd1c650129b56f2'}},'duration': 0.0045299530029296875,'error': None,'name': 'Clam AntiVirus Scanner','platform': 'linux2','results': None,'status': 0,'type': 'antivirus','version': '0.99'}

And check the output.

7.3.3 Debug Ansible Provisioning

To debug errors while provisioning (same goes with deployment) with following typical command:

$ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.→˓vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant→˓playbooks/provisioning.yml

Example output:

TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so→˓that the environment value uses thefull variable syntax ('{{rabbitmq_users_definitions}}').This feature will be removed in a future release. Deprecationwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u→˓'brain'}) => {"failed": true, "item": {"password": "brain", "user": "brain", "vhost→˓": "mqbrain"}, "module_stderr": "", "module_stdout": "Traceback (most recent call→˓last):\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 302,→˓in <module>\r\n main()\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_→˓user.py\", line 274, in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/→˓ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 155, in get\r\n users =→˓self._exec(['list_users'], True)\r\n File \"/tmp/ansible_wKXoO5/ansible_module_→˓rabbitmq_user.py\", line 150, in _exec\r\n rc, out, err = self.module.run_→˓command(cmd + args, check_rc=True)\r\n File \"/tmp/ansible_wKXoO5/ansible_modlib.→˓zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n File \"/usr/lib/→˓python2.7/posixpath.py\", line 261, in expanduser\r\n if not path.startswith('~→˓'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg":→˓"MODULE FAILURE", "parsed": false}

You could first increase ansible verbosity by adding -vvv option (-vvvv on windows for winrm debug), it will helpis the problem is linked to arguments.

$ ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key --→˓inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u→˓vagrant playbooks/provisioning.yml

TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************



task path: /home/alex/repo/irma-ansible/roles/Mayeu.RabbitMQ/tasks/vhost.yml:13[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks

→˓so that the environment value uses the fullvariable syntax ('{{rabbitmq_users_definitions}}').This feature will be removed in a future release. Deprecation warnings can bedisabled by setting deprecation_warnings=False in ansible.cfg.<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o

→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.→˓ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" && echo ansible-tmp-→˓1468570550.09-211613386938202="` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09-→˓211613386938202 `" ) && sleep 0'"'"''

<127.0.0.1> PUT /tmp/tmpiysJ6l TO /home/vagrant/.ansible/tmp/ansible-tmp-→˓1468570550.09-211613386938202/rabbitmq_user

<127.0.0.1> SSH: EXEC sftp -b - -C -o ForwardAgent=yes -o Port=2222 -o→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 '[127.0.0.1]'

<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o

→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 -tt 127.0.0.1 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"→˓'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rbeeckncuxenewcwkayivqiwvarchlrd; LANG=fr_FR.UTF-→˓8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /home/vagrant/.ansible/→˓tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user; rm -rf "/home/vagrant/.→˓ansible/tmp/ansible-tmp-1468570550.09-211613386938202/" > /dev/null 2>&1'"'"'"'"'"'"→˓'"'"' && sleep 0'"'"''

failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user→˓': u'brain'}) => {"failed": true, "invocation": {"module_name": "rabbitmq_user"},→˓"item": {"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr→˓": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/→˓ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 302, in <module>\r\n→˓main()\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 274,→˓in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/ansible_Qo3lZl/ansible_→˓module_rabbitmq_user.py\", line 155, in get\r\n users = self._exec(['list_users→˓'], True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line→˓150, in _exec\r\n rc, out, err = self.module.run_command(cmd + args, check_→˓rc=True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_modlib.zip/ansible/module_utils/→˓basic.py\", line 1993, in run_command\r\n File \"/usr/lib/python2.7/posixpath.py\",→˓ line 261, in expanduser\r\n if not path.startswith('~'):\r\nAttributeError:→˓'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed→˓": false}

In this particular case, verbose doesnt add much information as the problem is linked to ansible scripts. Lets go onelevel deeper so. Ansible output the temporary script executed on guest (highlighted in previous code block), but deleteit just after execution. To further debug it we will set ansible to keep remote files and the debug session will now takesplace inside the guest.

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv --private-key=~/.vagrant.d/→˓insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/→˓vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml

7.3. How to debug 59


in debug log get the temporary ansible path to remote script:

/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/→˓rabbitmq_user

Log in to remote machine and go to the temporary ansible dir. Explode the compressed script and run it locallly:

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ lsrabbitmq_user

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python→˓rabbitmq_user explodeModule expanded into:/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls debug_→˓dir/ansibleansible_module_rabbitmq_user.pyargs

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python→˓rabbitmq_user executeTraceback (most recent call last):

File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible_module_rabbitmq_user.py", line 302, in <module>

main()File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

→˓dir/ansible_module_rabbitmq_user.py", line 274, in mainif rabbitmq_user.get():

File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible_module_rabbitmq_user.py", line 155, in get

users = self._exec(['list_users'], True)File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

→˓dir/ansible_module_rabbitmq_user.py", line 150, in _execrc, out, err = self.module.run_command(cmd + args, check_rc=True)

File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible/module_utils/basic.py", line 1993, in run_command

args = [ os.path.expandvars(os.path.expanduser(x)) for x in args if x is not None→˓]File "/usr/lib/python2.7/posixpath.py", line 261, in expanduserif not path.startswith('~'):

AttributeError: 'list' object has no attribute 'startswith'

You could now add debug to source files and properly understand where the problem is. In our example case, it is anansible problem related to module_rabbitmq_user present in 2.1.0.0 see github PR

7.4 How to migrate

Note: If you need help to connect to your box through ssh, see vagrant FAQ

This part is only useful to someone willing to manually upgrade from an older version of IRMA.


https://github.com/ansible/ansible-modules-extras/pull/2310


7.4.1 Install alembic

$ sudo su deploy$ cd /opt/irma/irma-frontend/current$ ./venv/bin/pip install alembic$ export PYTHONPATH=.:$PYTHONPATH$ alembic history

430a70c8aa21 -> eb7141efd75a (head), version 1.3.02cc69d5c53eb -> 430a70c8aa21, version 1.2.1<base> -> 2cc69d5c53eb, DB revision creation

7.4.2 from 1.2.1 to 1.3.0

Fix nginx configuration

Introducing multiversion API means python code should receive the api version parameter. in file /etc/nginx/sites-available/irma-frontend.conf replace:

rewrite ^/api/v1/(.+) /$1 break;

by:

rewrite ^/api/(.+) /$1 break;

and restart nginx

Migrate Database

First you should tell alembic you are at version 1.2.1:

$ ./venv/bin/alembic stamp 430a70c8aa21

then upgrade model and data:

$ ./venv/bin/alembic upgrade head

Regenerate IHM

to regenerate IHM do the following:

$ sudo su deploy$ cd /opt/irma/irma-frontend/current/web$ ./node_modules/.bin/bower update$ ./node_modules/.bin/gulp dist

Its done.

7.5 API documentation

There is a dynamic documentation for IRMA API available on your instance

7.5. API documentation 61

http://172.16.1.30/swagger


It allows you to read documentation but also try request and see server response. Give it a try.

You could see detailed information about one specific API route:

and by clicking on the Try it button, see the server response:



7.6 Connect to a vagrant box through ssh

If you don’t already have it download vagrant insecure_private_key

$ wget https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant -O→˓insecure_private_key

Then change rights on the key otherwise ssh will complains and connect to your vagrant box

$ chmod 700 insecure_private_key$ ssh [email protected] -i insecure_private_key

7.7 Enable SSL using OpenSSL in ansible scripts

If you want to activate SSL on the frontend server, you’ll need:

• modify frontend_openssl variables in group_vars/frontend:

frontend_openssl: True # Default is falsefrontend_openssl_dh_param: # put the DH file locationsfrontend_openssl_certificates: [] # an array of files {source, destination}

# to copy to the server

• Uncomment (and customize) the nginx_sites variable in the group_vars/frontend, a commented example is avail-able.

Then, provision or re-provision your infrastructure. Ansible will only change file related to OpenSSL and Nginxconfigurations.

7.6. Connect to a vagrant box through ssh 63


7.8 Speed up your Vagrant VMs

Install this softwares:

• vagrant-cachier (more info on vagrant-cachier)

$ vagrant plugin install vagrant-cachier

• vagrant-vbguest (more info on vagrant-vbguest)

$ vagrant plugin install vagrant-vbguest


https://github.com/fgrehm/vagrant-cachier

https://github.com/dotless-de/vagrant-vbguest

CHAPTER 8

Resources

• Project website

• IRC (irc.freenode.net, #qb_irma)

• Twitter (@qb_irma)

65

http://irma.quarkslab.com

irc://irc.freenode.net/qb_irma

https://twitter.com/qb_irma


66 Chapter 8. Resources

CHAPTER 9

Screenshots

9.1 Command Line Interface

A sample script can be found in frontend repository. Add your own frontend address before testing it.

67


9.2 Web Interface

Some screenshots of the irma user interface shipped with frontend package.

68 Chapter 9. Screenshots


9.2. Web Interface 69


9.2. Web Interface 71

irma documentation - media. · pdf fileirma documentation, release 2.0.4 irma is an...

Documents