minion documentation · minion documentation, release 0.4 the last command will create a virtualenv...
TRANSCRIPT
Minion DocumentationRelease 0.4
Mozilla
January 07, 2014
Contents
1 Getting Started 31.1 Pre-requisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Installing Minion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Running Minion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Create Minion Admin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Explore the frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.6 Install New Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.7 Moving from beginngers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Configure Minion 72.1 Hostname Whitelist and Blacklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Configure Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Configure Frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Using Frontend 93.1 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 Scan Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.4 Issue Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.5 Admin: Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.6 Admin: Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.7 Admin: Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.8 Admin: Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.9 Admin: Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.10 Admin: Invites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Install Minion 154.1 Install Minion Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.2 Install Minion Frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5 Inside Minion 175.1 Minion’s mission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
6 Developing Plugins 216.1 How Minion discovers new plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.2 Minion Plugin Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.3 Import plugin class in __init__.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236.4 Plugin Template Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
i
7 Using setup.sh 257.1 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.2 Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.3 Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
8 Contribute 27
9 Developers 29
10 Indices and tables 31
ii
Minion Documentation, Release 0.4
Minion is a security testing framework built by Mozilla to bridge the gap between developers and security testers. Todo so, it enables developers to scan their projects using a friendly interface.
For more information see https://wiki.mozilla.org/Security/Projects/Minion
Source code are on Github: see minion-backend and minion-frontend.
Contents 1
Minion Documentation, Release 0.4
2 Contents
CHAPTER 1
Getting Started
This document shows how to get started with Minion by getting your own Minion installation for the first time.
1.1 Pre-requisites
• Minion has been developed primarily on Ubuntu 12.04 and 13.04 so running Minion on other systems is not aguarantee
• Python 2.7+ (not Python 3 compatible)
Finally, we need these dependencies fulfill:
sudo apt-get updatesudo apt-get install git build-essential python-virtualenv python-dev rabbitmq-server mongodb-server curl libcurl4-openssl-dev
Note: It is helpful to assume the followings for the purpose of this guide:
1. Minion’s Python dependencies will reside in a virtualenv
2. Minion and plugin repositories will reside under /home/user/ directory.
3. Point 1 and 2 are not required for production, but for the purpose of this tutorial.
1.2 Installing Minion
Minion has two components: minion-backend and minion-frontend. To get both installed, the quick way is to usesetup.sh. This script resides in a different repository: https://github.com/mozilla/minion.
Note: If you want to set up the backend and the frontend individually (suitable for production), please refer to InstallMinion Backend and Install Minion Frontend.
$ git clone https://github.com/mozilla/minion.git$ cd minion$ ./setup.sh clone$ ./setup.sh develop
By cloning we clone both the backend and frontend to the current directory (which in this case we assume it to be/home/user/minion directory.
3
Minion Documentation, Release 0.4
The last command will create a virtualenv called env under /home/user/minion. This virtualenv is shared bythe backend and frontend (this is not the best practice, but for the purpose of getting started this is enough). Thedevelop command will then source into this virtualenv and run python setup.py develop on the backendand the frontend to get all the dependencies fulfill.
There is also ./setup.sh install which performs python setup.py install. This is better for produc-tion. To learn more about the usage of the script, please refer to Using setup.sh.
1.3 Running Minion
As aforementioned, Minion is broken down into backend and frontend which are both using Flask web framework.Minion backend also runs three celery workers which are used to communicate between plugin scanning processes.
Let’s have five terminals opened. Each terminal will cd minion so that the current working directory is/home/user/minion. Then for each terminal, run one of the following commands:
$ ./setup.sh run-backend$ ./setup.sh run-frontend$ ./setup.sh run-plugin-worker$ ./setup.sh run-state-worker$ ./setup.sh run-scan-worker
By default, the backend runs on 127.0.0.1:8383 while the frontend runs on 0.0.0.0:8080; you can visitthe frontend by going to http://<vm-ip-address>:8080/ in your browser assuming that your VM IP isaccessible to your host computer.
If you wonder how these commands actually work, please refer to Using setup.sh.
To configure Minion, please refer to Configure Minion.
1.4 Create Minion Admin
Almost there! The very last bit is to create a Minion super user. To do this, the backend comes with a handy scriptcalled minion-db-init.
$ cd minion$ cd minion-backend/scripts$ ./minion-db-init
Enter the administrator’s Persona email:Enter the administrator’s name:Do you want to import some test sites into minion? [y/n]
Enter an email that has been registered as Persona email account. If you don’t have one or you are new to Persona,please check out persona before proceeding.
You don’t have to provide a real name, but this is the name we use in formal information such as sending invitation toa friend to join Minion. Mozilla does not collect data from self-hosted Minion if you care about your privacy.
You also don’t need to import test sites. By test sites we mean security testing sites. These sites are developed andfree to security testers for testing security tools. We encourage new users to import them when learning about Minion.You can always delete them later (or start from scratch when you deploy a production version).
4 Chapter 1. Getting Started
Minion Documentation, Release 0.4
1.5 Explore the frontend
I have a separate page to guide you how to use the frontend. Please check Using Frontend.
1.6 Install New Plugins
Plugins are essentials. As a new Minion owner, you should try installing new plugins. We have a separate guide onthis topic, so please check install_plugins.
1.7 Moving from beginngers
• Configure Minion
• Inside Minion
• Developing Plugins
• Using setup.sh
• Contribute
1.5. Explore the frontend 5
Minion Documentation, Release 0.4
6 Chapter 1. Getting Started
CHAPTER 2
Configure Minion
Minion backend and frontend can be configured after installation. This document will explain how to configure them.
As a convention, Minion will look at /etc/minion/ and /home/user/.minion. In the second case, the userhome directory is the user that runs Minion backend server.
2.1 Hostname Whitelist and Blacklist
As of Minion 0.3 release, Minion will blacklist the following IP addresses from scanning.
’10.0.0.0/8’,’127.0.0.0/8’,’172.16.0.0/12’,’192.168.0.0/16’,’169.254.0.0/16’
You can check the latest list from https://github.com/mozilla/minion-backend/blob/master/minion/backend/utils.py.
The effect of this is that Minion will refuse to scan any target site whose hostname falls in one of the ranges. Forexample, when Minion resolve the hostname localhost to 127.0.0.1, Minion will abort the scan because it isblacklisted.
To configure the blacklist and whitelist, you can supply a file called scan.json in either /etc/minion/ or/home/user/.minion/.
{"whitelist": [
"192.168.0.0/16","127.0.0.1"
]}
In this configuration, we allowed scanning LAN network and localhost. This is useful when you are testing your ownweb application from home. However, 172.16.0.0/12 range is still restricted from scanning.
You can supply your own blaclist as well.
{"whitelist": [
"192.168.0.0/16","127.0.0.1"
],"blacklist": [
7
Minion Documentation, Release 0.4
"foobar.com"]
}
In this example, foobar.com is not scannable. When we specify our own blacklist, we replace the default one entirelywith our own. So we can omit the whitelist in our example.
2.2 Configure Backend
Here is the default configuration for the backend server (see https://github.com/mozilla/minion-backend/blob/master/minion/backend/utils.py)
{’api’: {
’url’: ’http://127.0.0.1:8383’,},’celery’: {
’broker’: ’amqp://[email protected]:5672//’,’backend’: ’amqp’
},’mongodb’: {
’host’: ’127.0.0.1’,’port’: 27017
},’email’: {
’host’: ’127.0.0.1’,’port’: 25,’max_time_allowed’: 3600 * 24 * 7 # seconds in 7 days
}}
To configure the backend, supply all the options in a file called backend.json at either /etc/minion or/home/user/.minion.
The api/url is the full authority (hostname and port) of the backend server.
The max_time_allowed determines the life time of an invitation; by default it will remain valid for seven days.
2.3 Configure Frontend
The frontend is much simpler.
{’backend-api’: {
’url’: ’http://127.0.0.1:8383’}
}
If the backend server is on a different server, then put this configuration in a file called frontend.json at either/etc/minion or /home/user/.minion.
8 Chapter 2. Configure Minion
CHAPTER 3
Using Frontend
Minion’s frontend is separate from the backend repository. Here we show how the frontend looks.
3.1 Login
Minion’s frontend is authenticated using Persona. You can replace Persona if you want. The backend is authenticationagonistic; it is the job of the frontend web service to verify that given user has permission to access a resource. Weexplain this in Inside Minion.
9
Minion Documentation, Release 0.4
If you are new to Persona, please check persona to find out how to make your email address into a Persona-capableemail address.
3.2 Dashboard
The screenshot belows is the first thing a Minion user will see after logging in. The current logged in user can onlysee the sites that he or she has access to; the user must be in a group that the site is also in. We will explain this laterin the admin section.
The dashboard shows the progress of the newest scan (if any), the number of issues discovered by the newest scan, alink to view the detail of the newest scan and a button to trigger a scan.
The Scan History tab shows all the scans performed on the site. The All Issues tab lists issues of the newest scan foreach site the user has access to.
If the user is in multiple groups, the dashboard also provide a drop-down menu on the upper mid-right corner to switchthe group view, as shown in the screenshot below. In this case this current logged in user belongs to two groups.
3.3 Scan Detail
The scan detail page lists all the issues with a short title. If the scan failed it will list the reason.
10 Chapter 3. Using Frontend
Minion Documentation, Release 0.4
3.4 Issue Detail
Every issue has a detail page that describes the problem, possible mitigation and a list of references to look up.
3.5 Admin: Users
This panel lists all the users in Minion. As with all other panels in the rest of the administration dashboard, there is afilter and an in-page search box to find users.
3.4. Issue Detail 11
Minion Documentation, Release 0.4
3.6 Admin: Sites
This panel lists all the sites that have been added to this Minion instance. Currently only admin users canadd sites and site url format is also restricted. As of 0.4 release, Minion can only accept url that lookslike http[s]://<subdomain>*.<domain>. For example, while 192.168.1.100, foobar.com andfoo.bar.com are valid, the frontend will not accept paths so that foobar.com/project/ is not acceptable.
There is a long standing ticket (https://github.com/mozilla/minion-frontend/issues/100) to loosen up this restriction inthe future release of Minion.
Note: Also, since 0.3 release there is a hostname blacklist and whitelist mechanism added to Minion. Hostname like127.0.0.1 is blocked by Minion by default. To whitelist and/or blacklist what hostname can be scanned, please refer toHostname Whitelist and Blacklist.
Every site can have one or more attack plan. An attack plan is a JSON configuration which specifies the type of pluginthe attack will use. Every site belongs to zero or more groups and only the users in those groups can scan the site. Asof 0.4 release, you must add the site first and then add the site and user to a group in the Group tab.
3.7 Admin: Groups
As aforementioned in the previous section, once a site has been added, it should be added to a group by going to theGroups panel. Every group contains a set of users and a set of sites; only these users can access these sites. Bob’spersonal blog can belong to two groups named Group A and Group B. As long as Alice is in one of these two groups,she can too scan Bob’s personal blog on Minion.
By pressing the Edit button, the user is redirected to an editor page.
12 Chapter 3. Using Frontend
Minion Documentation, Release 0.4
3.8 Admin: Plugins
A Minion plugin is basically a Python class that invoke a security scan tool. This scan tool can be as simple as a fewlines of Python code directly in the class or invoking an external exectuable binary (which can be a simple bash scriptor as complex as OWASP ZAP).
Minion’s backend actually ships with some simple Python plugins called basic plugin. In the screenshot above, theyare under the namespace minion.plugins.basic. These plugins check HTTP header and robots.txt. Other plugins arenot distributed with the backend. You must install them indivdually (although the backend repository has the sampleconfiguration file for zap, skipfish and nmap).
3.8. Admin: Plugins 13
Minion Documentation, Release 0.4
To learn more about how to install plugins, please check install_plugins. For plugin development, please referto Developing Plugins.
3.9 Admin: Plans
Plans are basically JSON blob/document that specifies what plugins to use (and their configurations). You can have aplan that utilizes the basic plugin and ZAP plugin.
3.10 Admin: Invites
If you want to invite another person to use your Minion, you can send them an invitation. You can add, resend, orcancel invitation. You can also check whether the invitation is accepted, declined or expired.
When a user is invited, the account is created, but the account is suspended until the user has accepted the invitationand is authenticated by logging in using a Persona address. If your friend has two email addresses and you invite yourfriend by using the non-Persona email address, your friend can accept the invitation and log in using his or her Personaemail address. Minion will take care of the transition (replacing non-Persona email address in the database which hisor her Persona account).
14 Chapter 3. Using Frontend
CHAPTER 4
Install Minion
Note: If you are new to Minion, you probably want to check out Getting Started instead. This document is for userswho want to know how to set up Minion components individually (possibly for deployment reason).
4.1 Install Minion Backend
$ virtualenv env$ source env/bin/activate$ git clone https://github.com/mozilla/minion-backend.git$ cd minion-backend$ python setup.py develop
First, you don’t have to use virtualenv, but we recommend do it. Secondly, python setup.py develop is reallyuseful for both development and production. It is up to you to decide which one works best for your deployment.
Next, we just need to launch the backend server and celery workers. There is a folder called scripts at the rootdirectory of the minion-backend repository. When we run setup.py, the setuptool will make all the scripts underthe minion-backend/scripts/ available to the shell.
So if you type minion- and try auto-completion, you will see minion-backend-api,minion-plugin-workers and etc. Let’s launcht server and the workers.
$ minion-backend-api$ minion-plugin-worker$ minion-scan-worker$ minion-state-worker
The backend accepts options. Check -h. By default the server listens on 8383 and is a localhost service.
Note: If you use virtualenv in the first place, then these “scripts” are only available if the shell is sourced into thevirtualenv. In other words, if you open a new terminal and then try minion- with auto-completion, the chance is youwon’t see anything. If you install Minion without virtualenv, these scripts will be availiable to the $PATH.
4.2 Install Minion Frontend
Similar to the backend (see above):
15
Minion Documentation, Release 0.4
$ virtualenv env$ source env/bin/activate$ git clone https://github.com/mozilla/minion-frontend.git$ cd minion-frontend$ python setup.py develop$ minion-frontend -a 0.0.0.0
If you intall the backend on the same server as the frontend, and you want to use virtualenv, you don’t have to sharethe same virtualenv with the backend. You can create a separate virtualenv (which is a good practice) for the frontendand one for the backend.
Also, minion-frontend accepts options. Try minion-frontend -h to see the options. We specify -a to be0.0.0.0 so that we can access the frontend from the browser by going to http://<vm-ip>:8080. By default,the frontend listens on 8080 and is a localhost service. By changing to 0.0.0.0 we avoid having to port forwardor setting up a proxy server like Nginx. But of course, you want to keep the frontend a localhost service in realdeployment...
16 Chapter 4. Install Minion
CHAPTER 5
Inside Minion
In this document, we will look at the internals of Minion. Many of the points below have been discussed previouslyat:
• Introducing Minion
• Writing Minion Plugins
While the blog posts may be outdated, they are useful to look at.
5.1 Minion’s mission
Minion is developed as an open source project by the Mozilla Security Assurance team. The goal is to enable devel-opers and security testers to bring web application security into a continuous testing cycle. To do this, both developersand testers are given an easy to use dashboard that lists all the sites they need to scan, detail scan summary page anddetail issue page to explain vulnerability. Instead of learning a new scan tool, users can write Minion plugins to calltheir favorite scan tools, and Minion will spawn the tool as a new process, scan the target website, and return the resultsback to the user.
5.2 Architecture
At a high level there are three major components in Minion: Plugins, Task Engine, and Front End.
In principle, the backend of Minion consists of a task engine and a set of plugins. The Front End exposes a userinterface for both regular users and administrators to manage configurations and to see scan results.
5.2.1 Technology stack
Before we divde into what each component does, let us look at the technology stack we are using in Minion:
• We build the backend and frontend heavily in Python, using the Flask web framework for building an APIsserver.
• The backend’s task engine uses Twisted for plugin execution.
• We use RabbitMQ to queue and transfer messages between executing plugin and the backend’s task engine.
• The Front-end has a layer of wrapper APIs serving with Flask web framework.
• We create the front end interface using angular.js and Twitter Bootstrap by making http requests to our frontend’s APIs (which in turns make requests to the backend’s API server).
17
Minion Documentation, Release 0.4
• We use MongoDB as our data store.
• By default, our version of Minion is using Persona for authentication.
This stack corresponds to the diagram shown below.
5.2.2 Plugins
Minion plugins are light-weight wrappers that perform tasks such as configuring, starting, stopping a plan, and accepta set of callbacks to notify the caller that information is available. Minion ships with several primitive plugins knownas basic plugin. This group of plugins checks to see whether the target URL is accessible, examines various HTTPheaders and verifies whether robots.txt exists and correct.
Two base classes for plugins are provided in the Minion backend to get developers started:
• BlockingPlugin provides the basic functionality to support a plugin that performs a task, and reports it’s com-pletion state at the end. This is suitable for creating straightforward plugins directly within Python.
• ExternalProcessPlugin provides the functionality required to kick-off an external tool, and the class providesthe basis for several other extensions, especially those that wrap existing security tools.
Besides the primitive basic plugins, Mozilla developers have created several plugins. See community_plugins_label.
5.2.3 Task Engine
Minion Backend and Minion Task Engine are synonym in this docuemtnation. The Task Engine provides the corefunctionality for managing users, groups, sites, scans, and results within the Minion platform. Acting as a central hub,the Task Engine maintains a register of available plugins, provides facilities for creating and modifying plans, andmanaging user access to Minion, including which sites they can scan.
To execute a scan on a target URL, one or more plan must be defined for the target URL and a scan is initiated bypicking a specific plan. In the nutshell, a plan is a JSON document that specifies some information about what the plandoes, and a sequence of plugins to invoke (and such sequence is called a workflow). An example is shown below:
{"name": "Fuzz and Scan","description": "Run Skipfish to fuzz the application, and perform a ZAP scan.","workflow": [
{"plugin_name": "minion.plugins.skipfish.skipfish.SkipfishPlugin","description": "","configuration": {}
},{
"plugin_name": "minion.plugins.zap.zap_plugin.ZAPPlugin","description": "Run the ZAP Spider and Scanner","configuration": {
"scan": true}
}]
This plan will invoke Skipfish and ZAP. The configuration for Skipfish plugin is left empty to use the default options.Some plugins have required and optional configuration parameters for users to specify.
18 Chapter 5. Inside Minion
Minion Documentation, Release 0.4
5.2.4 Front End
As we stated in the technology stack section, the front end server is a Flask web application, while the user interface iscreated using AngularJS. In simple terms, users do not make direct API calls to the backend. In the nutshell, the taskengine has very very little access control built-in. The front end is responsible for creating the ACL by wrapping APIrequests to the task engine in the front end’s API. AngularJS makes calls to the front end’s API endpoints.
This looks confusing and unncessary for newcomer, but the main advantage is that we can re-engineer the front endin anyway we want with little to zero impact to the task engine. For example, someone swap out the entire frontend and our Persona authentiation with their own front end implementation and authentication model (e.g. LDAPauthentication).
5.2. Architecture 19
Minion Documentation, Release 0.4
20 Chapter 5. Inside Minion
CHAPTER 6
Developing Plugins
Minion plugins are Python classes that talks to a security aessement tool. This tool can be pure Python code inside theclass doing HTTP header check (which is what the basic plugins do), or invoke an external executable binary. It canbe as simple as a few lines of perl script or as complex as OWASP ZAP.
6.1 How Minion discovers new plugins
If we look at minion-zap-plugin’s source code structure closely, we notice the layout of the package looks like this:
$ tree minion-zap-plugin
minion-zap-plugin/|-- minion| |-- __init__.py| |-- __init__.pyc| ‘-- plugins| |-- __init__.py| ‘-- zap| |-- config.xml.example| |-- __init__.py| |-- reference.py| |-- zap_plugin.py|-- README.rst|-- setup.py
Every plugin must fall under the minion.plugins package namespace. This is how the backend detects an ex-istence of a plugin. The third level in minion-zap-plugin is a directory called zap. This is the namespace of theplugin itself. In addition, the plugin class must be a subclass of AbstractPlugin and the following class memberattributes:
** PLUGIN_NAME: the name of the plugin displayed to the frontend user
** PLUGIN_WEIGHT: level of resource required to launch this plugin (light, medium, heavy)
The registry code is found in base.py under the views directory.
Whenever a plugin is installed, the backend server and all the celery workers must be restarted.
6.2 Minion Plugin Classes
Now that you know how Minion discovers a new plugin, we can examine plugin classes.
21
Minion Documentation, Release 0.4
The first thing a plugin author needs to understand is the basics of Twisted. Minion’s Task Engine uses Twisted forrunning plugins. We recommend plugin authors go over An Introduction to Asynchronous Programming and Twistedbefore jumping into writing plugin. We will only cover relevant information to get you understand how a Minionplugin works.
6.2.1 AbstractPlugin class
The AbstractPlugin class implements a set of methods and attributes that a plugin should define. It also pro-vides additional methods that need not be override by plugin authors. The AbstractPlugin should be thebase class for any plugin you want to write. Minion ships with two classes of plugins (BlockingPlugin andExternalProcessPlugin) inherit from AbstractPlugin and other plugins Minion developers have devel-oped so far are inheriting from one of the two classes. We will get into them in the later section; let’s focus onAbstractPlugin.
Remember how Minion discovers and registers new plugin? The registry code expects the plugin to be found in theminion.plugins namespace, and in addition the plugins must have three class methods defined: name, version,and weight.
The AbstractPlugin implements these class methods with a default value. For example, if you don’t define name,it will use the plugin’s class name as the name of the plugin. But it is always a good idea to define them yourself.
To actual do work, a plugin should implement do_configure, do_start and do_stop methods since theAbstractPlugin leaves them blank.
6.2.2 BlockingPlugin class
This class inherits from AbstractPlugin directly and is defined in base.py like AbstractPlugin.
The BlockingPlugin implements do_configure, do_start, and do_stop methods. Most of the Twsitedlogics are defined in the do_start method. The class simply expects a method called do_run and passes thismethod to the deferToThread which will return an asynchronous result that will return an actual result some timein the future. The nice deferToThread API has a pair of callbacks: success and failure. We simply add these twocallbacks to the defer result object and returns.
The magic of Twsited is the event-driven model, also known as reactor model. The idea is that an event loop will reactto an event, possibly executes the event, and then emits a callback when the event is finished.
You are probably wondering what happen to do_configure. This particular class of plugin does not expect anypre-processing configuration needed. As we study the source code, we will see that all basic plugins are instances ofBlockingPlugin. After all, they just need to connect to the target URL, examine a header or validate the robots.txt.
6.2.3 ExternalProcessPlugin class
This is another implementation of AbstractPlugin class and this class is useful for executing a pluginthat relies on external process such as ZAP or Skipfish. Instead of calling subprocess, we use Twsited’sreactor.spawnProcess to launch the external process.
Keen readers notice do_configure is also not defined. This is intentional. Every plugin that inherits from theExternalProcessPlugin will implement its own do_configure because different process is launched dif-ferently.
22 Chapter 6. Developing Plugins
Minion Documentation, Release 0.4
6.3 Import plugin class in __init__.py
When we revisit the structure of a plugin package, it looks like this:
minion-zap-plugin/|-- minion| |-- __init__.py| |-- __init__.pyc| ‘-- plugins| |-- __init__.py| ‘-- zap| |-- config.xml.example| |-- __init__.py| |-- reference.py| |-- zap_plugin.py|-- README.rst|-- setup.py
The third __init__.py should import the main plugin class. In the case of the zap plugin, it looks like this:
from zap_plugin import ZAPPlugin
If the file is left blank, Minion’s plugin discovery code will not be able to import ZAPPlugin.
6.4 Plugin Template Generator
Because developing a plugin with the right structure can be tedious and error-prone, there is a script to generate Minionplugin. The script lives here: https://gist.github.com/yeukhon/8309083
To use this script, download generate-plugin.py and config.ini. Then edit config.ini to fit your need (see sample-config.ini on the same page as an example).
6.3. Import plugin class in __init__.py 23
Minion Documentation, Release 0.4
24 Chapter 6. Developing Plugins
CHAPTER 7
Using setup.sh
This document describes using setup.sh script.
7.1 Options
-x option specify the root directory of where minion repositories and the virtualenv should reside. By default, this isthe current directory of the current terminal. This option is usable for all the commands.
-a option specify the address the web server (minion-backend and minion-frontend) runs on.
-p option specify the port the web server (minion-backend and minion-frontend) listens on.
7.2 Commands
Command Descriptionclone Clone minion-backend and minion-frontend to the location specify by -x option.develop Create and source into env virtualenv located next to the repositories at the location specify by -x
option. Then run python setup.py develop.install Similar to develop except runs python setup.py install.run-backend
Run minion-backend-api script distrubted in minionp-backend to start backend server.Default to -p 8383 and -a 127.0.0.1.
run-frontend
Run minion-frontend script distrubted in minionp-frontend to start frontend server. Defaultto -p 8383 and -a 127.0.0.1.
run-plugin-worker
Run minion-plugin-worker script distributed in minion-backend to start a celery worker torun a plugin.
run-scan-worker
Run minion-scan-worker script distributed in minion-backend to start a celery worker toinititaite and collect scan results.
run-state-worker
Run minion-state-worker script distributed in minion-backend to start a celery worker tokeep track of the state of scan.
7.3 Usage Examples
$ pwd/home/user/minion$ ./setup.sh clone$ ./setup.sh develop
25
Minion Documentation, Release 0.4
$ ./setup.sh run-backend$ lsenv minion-backend minion-frontend setup.sh README.md
This will use all the default options. It will clone minion-backend and minion-frontend to /home/user/minionand the backend is started on 127.0.0.1:8383.
$ pwd/home/user/minion$ mkdir projects$ lssetup.sh README.md projects$ ./setup.sh clone -x projects$ ls projectsminion-backend minion-frontend
In this session, an empty directory called projects is created at /home/user/minion. The -x op-tion is used and projects is specify. The clone command then clone the two repositories down into/home/user/minion/projects.
We can continue setting up Minion like this:
$ ./setup.sh develop -x projects$ ls projectsenv minion-backend minion-frontend$ ./setup.sh run-backend -x projects -p 1234$ ./setup.sh run-plugin-worker -x projects
Following the previous session, we install Minion by specifying where minion-backend and minion-frontend resides.The virtualenv env is created. Afterward, we start the backend server and specify that the virtualenv env is under thedirectory projects. We also specify the server should listen on port 1234 instead of the default 8383. We also specifythe location of the env when we run the plugin worker.
Note that -a and -p are only usable for the servers. Workers can use the -x.
26 Chapter 7. Using setup.sh
CHAPTER 8
Contribute
Minion is built using open source projects such as Python, AngularJS and Twsited, so we want to keep it open source.Furthermore, we believe that “secrecy does not buy additional security.” (Kerckhoff’s principle) Security testing isnot an exclusive privilege. Instead, security testing should be as agile as possible.
In order to make Minion feature richer and secure, we invite the community to contribute in any way they can. If youhave ideas to submit, or have issues to report, you encoruage to contact us via:
• Source code: https://github.com/mozilla/minion
• Email list: http://groups.google.com/group/mozilla-minion-dev
• Security/Projects/Minion/Roadmap
• We also use the #websectools channel on irc.mozilla.org
27
Minion Documentation, Release 0.4
28 Chapter 8. Contribute
CHAPTER 9
Developers
We are available on #websectools or on Github.
• Stefan Arentz (st3fan)
• Yeuk Hon Wong (yeukhon)
• Simon Bennetts (psiinon)
• Mark Goodwin (mgoodwin)
• Matthew Fuller
29
Minion Documentation, Release 0.4
30 Chapter 9. Developers
CHAPTER 10
Indices and tables
• genindex
• modindex
• search
31