nio documentation · •python 3.4 or greater •pip •virtualenv •git first, open a terminal in...

NIO DocumentationRelease 1.0.1

neutral.io

May 27, 2016

Contents

1 Table of Contents 11.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Building Your First NIO Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3 NIO Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.4 NIO Command Line Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.5 NIO log level component API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151.6 Component - Management Publisher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.7 REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.8 Communication Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181.9 Block Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.10 Block Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.11 Block Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231.12 Block Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

i

CHAPTER 1

Table of Contents

1.1 Getting Started

1.1.1 Requirements

• Python 3.4 or greater

• pip

• virtualenv

• git

First, open a terminal in your home directory and check that you have everything you need by running these commands:

python3 --versionpip3.4 --versionvirutalenv --versiongit --version

If those four commands don’t return anything, then follow the requirement links at the top before continuing. Once allthe commands return a valid version, we are ready to move on to System Setup.

For Windows Run the above commands in a Git Bash Terminal. To access this, make sure you select “Git BashHere” when installing git. Then go to any folder and right click -> Git Bash Here Before running thecommands, you’ll need to follow the directions for Windows listed below. Also, be sure to pay attention to theFor Windows notes throughout the tutorial.

Sometimes the windows installation can be a pain. For help see(https://docs.python.org/3.4/using/windows.html). In the simplest case, you’ll have to do the following:

• Install Python3.4 for Windows from the link above

• Go to Control Panel -> System -> Advanced System Settings -> Environment Variables

– In System variables select Path and hit Edit

– Append (do not delete any text that exists) the following text at the end of the line:;C:\Python34;C:\Python34\Scripts;

– hit OK until out of all configuration windows

• Create a python3 shortcut to work with the rest of this tutorial You may be able to skip this and justuse python instead of python3 for the rest of this tutorial

1

https://www.python.org/download/https://pip.pypa.io/en/latest/installing.htmlhttp://docs.python-guide.org/en/latest/dev/virtualenvs/http://git-scm.com/downloadhttps://docs.python.org/3.4/using/windows.html

NIO Documentation, Release 1.0.1

• Open cmd in Administrator Mode (Windows Key -> locate command prompt -> right clickcommand prompt -> Run as Administrator

• type: cd C:\Python34

• type: mklink python3.exe python.exe

Pro Tip: After modifying anything in the system (such as all the actions from the above section), you’ll need toclose and re-open your git bash window to see your changes reflected. Simply close the window, right click, andselect “git bash here” once more.

Congratulations, Windows users! Now you can verify your requirements are met by running the 4 commands atthe top of the page.

1.1.2 System Setup

Each nio project will be in its own directory. We will start by creating some directories for these:

cd ~mkdir niocd niomkdir projectsmkdir versions

1.1.3 Global Installation (no virtual environment)

Let’s start by installing nio version 1.5.3 globally (without the use of a python virtual environment). In most cases,this is the method you will want to use. If you’re installing n.io onto a device where you’ll need the ability toswitch between different versions for development/testing purposes, you may want to skip ahead to the intructions forinstalling with a virtual environment, but for almost all other cases, a global install is the way to go.

First we need to make sure our user has the proper permissions to install python packages via pip. This is doneby adding our user to a group that has write permissions to /usr/local/lib/python3.4/site-packages.This group is usually staff on most linux distributions. The command to do this on linux would be:

sudo usermod -a -G staff

And on OSX:

sudo dseditgroup -o edit -a -t user staff

Now we need to get the wheels for nio and nioext onto the device which n.io will be installed to. Once the wheels areon the device, navigate to the directory they have been copied to, and install them with these commands:

sudo pip3.4 install nio-1.5.3-py3-none-any.whlsudo pip3.4 install nioext-1.5.3-py3-none-any.whl

**Note: Depending on how pip was installed, the command may be slightly different. You can verify the commandto call pip by trying pip, pip3, or pip3.4 with the --version option. Also, if you have access to the web hostednio wheels, you can just use sudo pip3.4 install http:// and skip copyingthem locally to the device.

If you need to install using a virtual environment or you are using windows, follow along in the section below. Ifyou’re NOT installing to a virtual environment or using windows, we can now move on to Installing nio CLI.

2 Chapter 1. Table of Contents


1.1.4 Virtual Environment Installation

This section will guide you as to how n.io is installed contained within a python virtual environment. This keeps n.ioand all it’s dependencies isolated, and can be useful when you need the ability to switch between different versions ofn.io, such as in a development or testing environment. You can read more about virtual environments here.

cd ~/nio/versionsvirtualenv -p python3 1.5.3

For Windows The previous command will be: virtualenv -p C:/Python34/python.exe 1.5.3

You now need to activate your virtual environment. This will add a (1.5.3) to the beginning of your command line toindicate the virtual environment that you are using.

source 1.5.3/bin/activate

For Windows The previous command will be: source 1.5.3/Scripts/activate

When you’re done using nio, you can leave the virtual environment with deactivate. When using nio again, besure to activate the virtual environment first with source ~/nio/versions/1.5.3/bin/activate.

OK, now we can install nio. If you don’t have these wheels then you’ve got to make friends with someone who does!

pip install nio-1.5.1-py3-none-any.whlpip install nioext-1.5.1-py3-none-any.whl

For Windows pywin32 must also be installed:

easy_install http://sourceforge.net/projects/pywin32/files/pywin32/Build%20219/pywin32-219.win-amd64-py3.4.exe/download

Note: make sure you take note of the “bittedness” of your python installation. The link above provides thecommand to install pywin32 on a system running 64-bit python. If you are running 32-bit python, the commandwill be:

easy_install http://sourceforge.net/projects/pywin32/files/pywin32/Build%20219/pywin32-219.win32-py3.4.exe/download

The installation of nio is now complete! You can run the instance from a project directory with the run_nio com-mand. See Setting up a NIO Project for instructions on creating a project directory.

1.1.5 Installing nio CLI

We now install the nio CLI, a tool that takes care of common operations like creating projects, adding blocks, manag-ing dependencies, and running n.io itself. It is called the nio Command Line Interface (CLI).

pip install nio-cli

1.1.6 Setting up a NIO Project

To use the nio CLI you need git (a distributed version control tool) and a GitHub account with ssh access.

After getting your github account set up and adding your ssh key to it (click the links above if you need helpwith this), verify that you can connect to git with: ssh -T [email protected]. It should return something like “Hi! You’ve successfully authenticated, but GitHub does not provide shell access.”

If you have having trouble connecing, type this and then try again: eval $(ssh-agent -s) && ssh-add~/.ssh/id_rsa

To help you get started, we provide a project template which reflects the standard directory structure of a nio project.Use the nio CLI to create a new project from the template:

1.1. Getting Started 3

http://docs.python-guide.org/en/latest/dev/virtualenvs/https://git-scm.comhttp://github.comhttps://help.github.com/articles/generating-ssh-keyshttps://github.com/nio-blocks/project_template


cd ~/nio/projectsnio new cd

If you don’t have ssh access set up for GitHub then try using the https flags:

nio new --https

The first thing we’re going to need is some blocks. We provide a selection of open source blocks for your convenience, but, remember, nio is designed to make it easy for you todevelop custom blocks; more on this later. For now, let’s just get a group of blocks that we’ve categorized as util.

nio add util

Again, if you don’t have ssh access set up for GitHub then try using the https flags:

nio add util --https

Running nio

This part is simple. Navigate to your project directory (which you should still be in if you’ve been following along),and run the following command. If you’ve installed n.io to a virtual environment, make sure it’s active!

cd ~/nio/projects/nio server

If you see any ERROR messages you may have a problem. But for now lets ignore this one: NIO [ERROR][Discover] Failure loading module nioext.components.snmp.agent ImportError:Nomodule named ’pysnmp’. We won’t be using that anyway.

The previous command runs n.io with standard output to the console. This is excellent to ensure that your instance isrunning properly and troubleshoot dependency issues, but for the most part, you’ll want n.io running in the background.This is achieved by adding the -bg flag to the nio server command, like so:

nio server -bg

This method will simply return you to a command prompt, with no console message output, and will keep runninguntil the instance is terminated with nio co shutdown.

At this point we’re done running commands from the terminal, but we will be keeping an eye on these logs.

(Later, when you’re done, you’ll want to press ctrl-c to exit nio).

Creating your first service

nio has a web app that you can use to interact with a running nio instance. By default, the project_template runson 127.0.0.1:8181, so just visit and log in with the default administrator priviledges (username:Admin; password: Admin). You should see something like this:

open http://builder.n.io


http://builder.n.io


To demonstrate the most basic use of the web UI, we’ll design a service that generates nio signals automatically andlogs them to the nio logging. With the way the project template is configured, this means we will see the simulatedsignals logged to the console and to a log file for our service.

First, click the Add Service button that appears in the top-right corner of the center panel of the web UI. Let’s namethe service SimulateAndLog. When you’re done entering the service name, click Submit. At this point, your browserwindow should look something like this:

Now we can add a few blocks. The list in the left panel of the UI contains the list of block types currently loaded intonio. Scroll until you find the Simulator; click and drag it over to the SimulateAndLog grid. Name it TestSimulator andclick Submit. In the left panel, again, scroll to find the LoggerBlock, and drag it over to the grid. Name it TestLogger.

Click Save Service in the bottom right of the right panel (you should get a confirmation that the save was successful).

1.1. Getting Started 5


Click the Start Service button in the very bottom right of the UI, and watch the terminal where you executed run_nio.

You should see a bunch of log messages with information about starting and configuring the service, but no signalsget logged! This is because we didn’t connect the blocks in SimulateAndLog. Nio blocks can run in isolation until thecows come home, but they won’t communicate with each other until we explicitly connect them. Lets fix that.

First, stop the service (changes to a running service won’t be reflected in its behavior until it is restarted anyway).Next, connect TestSimulator to TestLogger. Click and drag from the dot on the underside of TestSimulator to the doton top of TestLogger.

Click Save Service and Start Service again. This time you should see signals logged to the console every second(check the timestamps).

Congratulations! You just built your first nio service!



Configuring blocks

Lets try changing our service by configuring the blocks to something other than the default behavior.

Click on the TestSimulator block to bring up its configuration in the right panel. Don’t worry too much about specificproperties here. To get familiar though, scroll down to the Interval section and change the number in the Secondstext box from 1 to 2; click Save Block. Now select TestLogger in the execution grid and use the drop-down menu tochange its Log Level and Log At to DEBUG, saving the block when you’re done.

Restart your service by clicking Stop Service and Start Service. This time you should see signals logged to theconsole every 2 seconds (check the timestamps).

Conclusion

Now that you’ve got a nio project with some blocks, try playing around with some of the other blocks. Change somemore configuration settings on TestLogger. What does Signal Count do?. Try putting a Counter between a Simulatorand a Logger. All the blocks have documentation on GitHub.

When you’re done with nio, go to the console where your logs are printing and press ctrl-c to exit nio.

1.2 Building Your First NIO Application

Before we get started, let’s go over some key concepts that are central to the NIO platform. In this document, we’lldiscuss Blocks, Signals, Services, the NIO Core, and steps you’ll need to take in order to build your first NIO app.

1.2.1 Terminology

• Instance - A running, well, instance of the NIO executable.

– Probably comprises multiple Services.

• Service - The main organizational component in NIO.

– Each service maintains a Block execution that defines connections between some number of blocks.

– Maintains its own, unique, configuration.

• Block - The fundamental unit of computation in NIO.

– Written in NIO-flavored Python, blocks either operate on Signals, interact with the outside world, orboth.

• Signal - The fundamental unit of data transfer in NIO.

• Modules - Uniform API’s designed to provide functionality common to many NIO applications without tyingdevelopers to any particular library of implementation.

• Property - A configurable field on a block or service.

• Core - Everything that NIO does that you don’t have to worry about.

– Manages block communication via signals, inter-process communication, block discovery, and muchmore.

1.2. Building Your First NIO Application 7

https://github.com/nio-blocks/util


1.2.2 Blocks

Here’s an example of the most basic possible block, one that accepts incoming signals and passes them along, un-touched:

from nio.common.blocks.base import Blockfrom nio.common.discovery import Discoverable, DiscoverableType

@Discoverable(DiscoverableType.block)class IDBlock(Block):

def process_signals(self, signals):self.notify_signals(signals)

With no modification, this block will be discovered by a correctly configured NIO service and could run as part ofits execution. The core delivers signals notified by other blocks to IDBlock.process_signals, where any necessaryprocessing is performed and, optionally, the signals are notified again.

For examples of more complex blocks making use of various modules and higher-order functionality, visit nio_blocks

Block methods

The following methods perform essential block functions and can be reimplemented in custom blocks. If overridden,it is almost always necessary to call the overriden method in the first line of the custom method.

• Blocks.configure - This method is called before the block is started. Note that unhandled exceptions here willcause the enclosing service to abort.

• Blocks.start - This method is called when the service enclosing is started by nio’s core. This is the place toschedule jobs, connect to external services, etc. Note that unhandled exceptions here will not effect the runningstatus of the enclosing service.

• Blocks.stop - This method is called when the enclosing service is stopped by nio’s core. Any persistent data,running threads, etc. should be cleaned up here.

• Blocks.notify_signals - Call this method from your block to forward an arbitrary list of Python objects to anyreceivers of your custom block. This is the primary output mechanism for blocks.

• Blocks.process_signals - This method is called when your block is configured as the receiver of another blockin the enclosing service. This is the primary input mechanism for blocks.

1.2.3 Services

Currently, we provide a single service class that has served our purposes quite nicely until now. However, we have leftopen the possiblility for developers to provide custom service classes for use in their nio projects. Here’s an example:

from nio.common.service.base import Servicefrom nio.common.discovery import Discoverable, DiscoverableType

@Discoverable(DiscoverableType)class BasicService(Block):

def start(self):# custom start logic here

So far, the default service has been sufficient for all our internal use cases; by design, much of the logic in your nioproject will reside at the block level.


http://github.com/nio-blocks/


1.3 NIO Expressions

1.3.1 ExpressionProperty

Given the event driven nature of most NIO applications, it can be difficult or impossible to completely describe thecontents of Block properties statically. For this reason, we provide a simple syntax for accessing Signal attributesat run-time. If the value of a Block property should reflect some value that may vary from Signal to Signal, simplydeclare that property an ExpressionProperty and grab its value inside Block.process_signals.

Here’s an example:

class MyBlock(Block):expr = ExpressionProperty(default='', attr_default=0)

def process_signals(self, signals):for s in signals:try:

value = self.expr(s)except Exception as e:

value = Noneself._logger.error("Block property 'expr' failed to evaluate: {}".format(e))

# Do some stuff with the value

It’s that simple. If the Signal under inspection is missing an attribute you ask for in the expression, the value ofattr_default (provided at property initialization) is used in its place. As with other NIO properties, the value of‘default’ will be evaluated if no expression is configured.

Any other errors in the expression will raise native exceptions whose handling is the responsibility of the Blockdeveloper.

Argument: attr_default

Unique to ExpressionProperty is the attr_default argument. In evaluating an expression, if the $ notation is used to getan attribute value from a signal, but the attribute does not exist on the signal, then the value attr_default will be usedin its place.

If attr_default is not specified on the ExpressionProperty, then an empty string is used.

attr_Default can be set to an Exception class. In this case, if the attribute does not exist on the signal, then thatException will be raised.

Handling Exceptions

An ExpressionProperty should always be evaluated inside of a try block. Invalid python syntax inside of the expressionwill raise an Exception and the block needs to handle that. This is also where you would handle the Exception raisedby attr_default if that is set to an Exception class.

1.3.2 Scripting Syntax

There are two types of results to consider when configuring an expression property:

• Immediate Value - A single python expression is evaluated and the result is returned without modifica-tion/stringification.

• String Interpolation - Snippets of Python code are evaluated and interpolated based on their position in anenclosing string.

String Interpolation

1.3. NIO Expressions 9


String interpolation in expression properties works similarly to string interpolation in other dynamic languages. Snip-pets of code are delimited as such:

...{{ }}...

and the result of each evaluation is inserted directly into the surrounding string, replacing the code snippet and enclos-ing braces. The code inside the braces can be pure python, if you like, as in:

"{{1 + 5}} dogs went to the park" -> "6 dogs went to the park"

Or you can use our special syntax for accessing signal properties:

# given a signal s s.t. s.v1 == 6"{{$v1}} dogs went to the park" -> "6 dogs went to the park"

Naturally, you can combine the two approaches, as in:

"My favorite Integer is {{$v1 if isinstance($v1, int) else 'not an Integer...'}}"

If you’d like to include a ‘$’ character in a string literal inside a code snippet, just escape it with a ‘’. Similarly,escaping the ‘}}’ or ‘{{‘ with a ‘’ causes them to be treated as strings rather than as delimiters. For example:

"Code snippets are delimited by \{{ and \}}" -> "Code snippets are delimited by {{ and }}"

Immediate Value

If you’d prefer that the result of your expression evaluation not be stringified, you can populate an expression fieldwith a single snippet enclosed in double-curlies:

# given a signal s s.t. s.v1 == 1, s.v2 == 'two', s.v3 == [3]"{{[$v1, $v2, $v3]}}" -> [1, 'two', [3]]

Raw Signal

You can access the signal itself (rather than just its attributes) with a lone ‘$’. As long as it is followed by a characterthat could not be the first character of a valid Python identifier, the ‘$’ evaluates to the incoming signal.

1.3.3 Examples

Here are some more examples that we find particularly illustrative:

"{{1 + 5}} dogs went to the {{'p' + 'ark'}}" -> "6 dogs went to the park"

# given a signal s s.t. s.v1 == {'who': 'Baron Samedi'}"{{$v1['who']}} and the Jets" -> "Baron Samedi and the Jets"

# given a signal s s.t. s.v1 raises AttributeError, s.v2 == 'Cogito'# and a default value of None"{{$v1 or $v2}} ergo sum" -> "Cogito ergo sum"

# given a signal s s.t. s.get_val() == 'foobar'"Opened it with a {{$get_val()}}" -> "Opened it with a foobar"

# given a signal s s.t. s has the attribute v1 but not v2"{{hasattr($, 'v1') and hasattr($, 'v2')}}" -> False



1.4 NIO Command Line Tools

1.4.1 nio

NIO exposes a rich API for interacting with running instances. We provide a robust browser-based graphical userinterface to same, but such interfaces are rarely well suited to automation of tasks and rapid context switching. For thisreason, we provide the nio command line tool, which allows users and administrators to start, stop, view, configure,and build blocks and services from a UNIX command line.

Here’s an example:

$ nio ls services TestPost+--------------+---------+| TestPost | |+--------------+---------+| auto_start | True || log_level | DEBUG || mappings | NULL || sys_metadata | NULL || type | Service |+--------------+---------+

nio is configurable by python .ini file. By default, it looks in $HOME/.config/nio-cli.ini under the section nio for ahostname, port, username, and password. The path to the .ini file is configurable by command-line option, but makesure you put all the nio options under the appropriate section. Here’s an example:

[DEFAULT]host = localhostport = 8181username = Userpassword = User

[nio]username = Adminpassword = Admin

nio exposes a number of sub-commands to perform various tasks. Each, and its syntax, is represented by one of thefollowing subsections.

new

This subcommand creates a fresh nio project from the official [project tempalte](https://github.com/nio-blocks/project_template). A new project folder will be made inside the current directory.

$ nio new project_name$ cd project_name$ lsblocks etc extras README.md nio.conf nio.env

add

This subcommand adds new block types to and existing nio project. By defualt they are pulled from the official [nioGitHub](https://github.com/nio-blocks) account.

This must be run from the root level of the project that you want to add the blocks too. One ore morse lists can beadded at a time.

1.4. NIO Command Line Tools 11

https://github.com/nio-blocks/project_templatehttps://github.com/nio-blocks/project_templatehttps://github.com/nio-blocks


$ nio add logger$ ls blockslogger$ nio add counter simulator$ ls blockscounter logger simulator

list (ls)

When called without a specific block or service name, this subcommand lists all the blocks or services loaded in theinstance along with a common subset of properties:

$ nio ls blocks+----------------+--------------+-----------+| name | type | log_level |+----------------+--------------+-----------+| Deb | Debouncer | ERROR || fil | Filter | ERROR || PostToMe | PostSignal | DEBUG || CountMe | Counter | ERROR || count | Counter | ERROR || FacebookPoster | FacebookPost | ERROR || TwitterPoster | TwitterPost | DEBUG || SignalLogger | LoggerBlock | DEBUG |+----------------+--------------+-----------+

When called with a block or service name, nio outputs a list of all the properties on that block or service:

$ nio ls blocks TwitterPoster+------------------------+----------------------------------------------------+| TwitterPoster | |+------------------------+----------------------------------------------------+| creds | || +-> oauth_token | XXXXXXXXXXXXXXXXXXX || +-> consumer_key | XXXXXXXXXXXXXXXXXXX || +-> oauth_token_secret | XXXXXXXXXXXXXXXXXXX || +-> app_secret | XXXXXXXXXXXXXXXXXXX || log_level | DEBUG || status | What the {{$foo}}? || type | TwitterPost |+------------------------+----------------------------------------------------+

To list the HTTP commands (with enumerated parameters) exposed by a particular block or service, simply appendthe ‘–cmd’ flag:

$ nio ls services TestPost --cmd+---------+------------------+| command | 0 |+---------+------------------+| log | phrase: (string) |+---------+------------------+

To view the block execution of a particular service (in tabular form), append the ‘–exec’ flag:

$ nio ls services SomeService --exec+--------------+--------------+| Output Block | 0 |+--------------+--------------+



| met | SignalLogger || CountMe | Deb || Deb | SignalLogger || PostToMe | CountMe |+--------------+--------------+

command (co)

This subcommand allows you to send commands to live instances. Because of the way blocks are referenced withinNIO, you must supply the target command, the name of the service, and (if you are commanding a block) the name ofthe block instance.

$ nio co start TestPost`http://localhost:8181/services/TestPost//start` was processed successfully

The syntax for adding parameters to commands is as follows:

$ nio co log TestPost SignalLogger --args 'phrase=foobar'

Passing the –auto (-a) flag to command tells nio to read command arguments from the command line.

If the command response body is non-empty, its contents are printed to stdout; otherwise, the terminal remains silent.

config (cfg)

This subcommand allows you to configure new and existing block and service instances while the NIO instance is live.Of course, a given service won’t load a new configuration until its next startup cycle, but any changes you make herewill hang around until then. Note that this is an interactive portion of the utility.

NB: If you want to automate configuration, it may be easier to make your updates directly via HTTP PUT requests.We may add a feature like this in future.

$ nio cfg services TestPost

log_level (select):Using current value: DEBUG

auto_start (bool): T

If the block or service you’re configuring holds an Object Property, each property held by that object is configured inturn:

$ nio cfg blocks TwitterPoster

creds (object):+->oauth_token (str):Using current value: XXXXXXXX+->consumer_key (str):Using current value: XXXXXXXX+->app_secret (str):Using current value: XXXXXXXX+->oauth_token_secret (str):Using current value: XXXXXXXX

status (str): "It's gonna rain"

1.4. NIO Command Line Tools 13


log_level (select):Using current value: DEBUG

nio interprets attempts to configure nonexistent resources as creation events. That is, the following sequence resultsin the creation of a block called “CriticalLogger”, which can subsequently be added to any service execution in thesystem.

$ nio cfg blocks AnotherLoggerNIOClient: NIO returned status 404type (str): LoggerBlocklog_level (select): CRITICAL+----------------------+-------------+| Name: CriticalLogger | |+----------------------+-------------+| log_level | CRITICAL || type | LoggerBlock |+----------------------+-------------+

Currently, NIO ships with only one service type (Service), but developers are free to add both new blocks and servicesas they wish.

update

The update subcommand command is very simple but very important. It compels a running NIO instance to rediscovereach of the block types in its argument list, allowing developers to perform live updates to block implementations. Forexample, after updating the implementation for the LoggerBlock:

$ nio update LoggerBlock

If all the block types are valid, standard out should remain silent.

NB: The current implementations of a running block will remain in memory until the enclosing service is restarted.

build (ln)

This final subcommand allows you to build NIO services from the command line by creating links between blocksthat are loaded into the sytem. First, let’s use ls to check out the execution of the TestPost service and view the list ofblocks available in the system.

$ nio ls services TestPost --exec+--------------+--------------+| Output Block | 0 |+--------------+--------------+| met | SignalLogger || CountMe | Deb || Deb | SignalLogger || PostToMe | CountMe |+--------------+--------------+

$ nio ls blocks+----------------+--------------+-----------+| name | type | log_level |+----------------+--------------+-----------+| count | Counter | ERROR || met | Metrics | ERROR || SignalLogger | LoggerBlock | DEBUG || PostToMe | PostSignal | DEBUG |



| Deb | Debouncer | ERROR || FacebookPoster | FacebookPost | ERROR || CountMe | Counter | ERROR || fil | Filter | ERROR || TwitterPoster | TwitterPost | DEBUG |+----------------+--------------+-----------+

Finally, we can use build to connect the outputs of CountMe and PostToMe to the input of another block!

$ nio build TestPost CountMe TwitterPoster+--------------+--------------+---------------+| Output Block | 0 | 1 |+--------------+--------------+---------------+| CountMe | Deb | TwitterPoster || met | SignalLogger | || PostToMe | CountMe | TwitterPoster || Deb | SignalLogger | |+--------------+--------------+---------------+

The build subcommand behaves somewhat similarly to the UNIX cp command in that it takes a series of n blockinstances of which the firs n-1 represent “source” blocks, while the *n*th block is the single sink.

Additionally, if you need to add a block fil to a service without connecting it to any other blocks (e.g. a block whichsimply serves an HTTP endpoint):

$ nio build TestPost fil+--------------+--------------+---------------+| Output Block | 0 | 1 |+--------------+--------------+---------------+| CountMe | Deb | TwitterPoster || met | SignalLogger | || PostToMe | CountMe | || Deb | SignalLogger | TwitterPoster || fil | | |+--------------+--------------+---------------+

In this case, fil has no receivers, and any that you add will appear starting from column 0 of the execution table.

Finally, if you’d like to remove blocks/block connections instead of create them, just append the -rm option to any ofthe above build invocations.

Voila! You’re now up and running with the NIO command line interface. Happy hacking!

1.5 NIO log level component API

1.5.1 Logger names retrieval

Logs can be retrieved through a GET call at the core or service level

1. core level uri: http://[ip address]:[port]/log

2. service level uri: http://[ip address]:[port]/log/service/[Service Name]

1.5.2 Changing log level

Log level can be changed through a POST or PUT call at the core or service level. Make a request to the correspondingURI with a PUT/POST Body formatted like this:

1.5. NIO log level component API 15

http://{[}iphttp://{[}ip


{"log_level": [New Log level],"logger_name": [Logger name]

}

Example:

PUT http://[ip address]:[port]/log/service/[Service Name]{

"log_level": "DEBUG","logger_name": "Custom Scheduler"

}

Note: For core and service level changes, if ‘logger_name’ parameter is missing in the body, the level is changedacross whole process

1.6 Component - Management Publisher

The Management Publisher Component hooks to service_status_change and notifies a ManagementSignal to the con-figured Publisher topics.

It also makes the Publisher available to services and blocks. Blocks can publish ManagementSignals withself.notify_mangement_signal(signal), as well as StatusSignals that will chage that status of the block.

1.6.1 Configuration

The management publisher topics are configurable. The Default value is: “topics”: “{“type”: “management”}”

Set confiugration by putting a management_publisher.cfg file in the project’s etc/ directory with a dictionary that sets“topics”.

{ “topics”: “{“type”: “mgmt”}” }

1.6.2 ManagementSignal

Subscribers to the management publisher can get the status with $status.name and the name of the service with $ser-vice.

Blocks that publish a ManagmentSignal or StatusSignal will also have a status that can be retrieved through $sta-tus.name, along with a $msg. As a standard block developers should put the notifing block’s name in $name and set$source to “Block”. The [RESTPolling block](https://github.com/nio-blocks/http_blocks#RESTPolling) is an exam-ple of that.

1.7 REST API

1.7.1 How to use

A great way to send / receive rest API calls is with the [Postman extension forChrome](https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en)

• set the url to http://IP_ADDRESS:PORT/COMMAND

• set correctly to either GET, POST, PUT, or DELETE


https://github.com/nio-blocks/http_blocks#RESTPollinghttps://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=enhttp://IP_ADDRESS:PORT/COMMAND


1.7.2 Commands

User authentication depends on the configuration of the nio project’s security settings.

1.7.3 /nio

GET

Get nio version, component versions and module versions.

1.7.4 /blocks

GET

Get all block configurations.

POST

Create a new block configuration.

Post body needs to include “type” and “name” where “type” is the block type and “name” is the name of your newblock configuration.

1.7.5 /blocks/{{block_name}}

GET

Get configuration of a block.

PUT

Update configuration of a block.

DELETE

Delete a block.

1.7.6 /blocks_types

GET

Get block types that are loaded into the nio instance.

1.7.7 /services

GET

Get all service configurations.

• include_pids: when set to ‘true’, process ids are included in response.

POST

Create a new service configuration.

Post body needs to include “type” and “name” where “type” is the serivce type (i.e. “Service”) and “name” is thename of your new service configuration.

1.7. REST API 17


1.7.8 /services/{{service_name}}

GET

Get configuration of a service.

PUT

Update configuration of a service.

DELETE

Delete a service.

1.7.9 /services/{{service_name}}/{{command_name}}

GET

Execute a command on a service.

Built-in commands are start, stop and status.

1.7.10 /services/{{service_name}}/{{block_name}}/{{command_name}}

GET

Execute a command on a running block inside a serivce.

1.7.11 /services_types

GET

Get service types that are loaded into the nio instance.

1.7.12 /shutdown

GET

Shutdown the nio instance.

1.8 Communication Module

1.8.1 Introduction

Nio provides a robust mechanism for effecting communication between processes. Based around ZeroMQ, Pub-lish/Subscribe, and the Broker Model, nio communication allows you to pass signals transparently between services,nio instances, or even physical devices (running nio, of course).



1.8.2 Configuration

• mode - communication mode

– zmq.manager

• matching - topic matching class

– nio.modules.communication.matching.default.DefaultMatching

• master - whether or not the master broker will run in this instance

– true/false

• bind_ip_address - the IP address of the master broker. if master is true, this should be the IP address of thehost machine. if master is false, this should match the “bind_ip_address” value of the communication modulefor the master.

• xpub_port - the port over which publishers are registered with the master broker.

• xsub_port - the port over which publishers are registered with the master broker.

• ip_address - the IP address of the host machine, regardless of whether master is true.

• poll_rate_ms - mysterioso

• max_port - upper bound of port range used for finding available local ports for publishers/subscribers

• min_port - lower bound

• connect_timeout - time to wait at start up when there is a communication failure due to ip address, port issues,etc.

– 0 - no wait, module fails right away

– > 0 - keep retrying for specified time unless IP Address is determined to be valid

– < 0 - keep retrying indefinitely unless IP Address is determined to be valid

1.8.3 Capabilities

When configured correctly with a single master instance, nio’s communication module can be used to create failure-robust, dynamic publish/subscribe networks. However, there are a few limitations:

• For two (or more) disparate machines to communicate (i.e. one pubs while the other subs), each must have anIP address which is accessible by the others. Within a local network, this shouldn’t be a problem, but, if one orboth of the IP addresses is behind a NAT, communication may not be possible.

• Two slave instances running without a master can create publishers and subscribers; however, messages will notbe transmitted between same until a master broker appears in the network. Even so, publishers and subscribersinstantiated before the master node is running will be detected and connected by the master broker when it starts.

Beyond that, just about any configuration of publishers and subscribers, local or remote, should work seamlessly,irrespective of the order in which publishers, subscribers, masters, or slaves are instantiated.

1.9 Block Development

1.9.1 Module logging

Sometimes your blocks will import python modules that use python logging. It is possible to configure the log levelof those to match your block. The recommended place for that is in the configure method.

1.9. Block Development 19


import requests

class NewBlock(Block):

def configure(self, context):super().configure(context)logging.getLogger('requests').setLevel(self._logger.logger.level)

Note that all imports of a python module in a service will share the same log level. So if mutliple blocks in one serviceset the log level to different levels, unknown behavior will occur as the log level will be chosen from whichever blockconfigures last.

1.10 Block Testing

Nio blocks are not meant to be run as standalone Python modules, so testing them can be a subtle and error proneprocess. We provide a couple of tools and best practices to make this a bit easier.

1.10.1 NIOBlockTestCase

We recommend that you use the NIOBlockTestCase when building your tests. It extends unittest.TestCase, and tiesinto all the same infrastructure you’re accustomed to using with the base class. However, NIOBlockTestCase alsoprovides a number of helpful methods for testing blocks. Here’s a real-world example straight from our internal blockrepositories:

from nio.util.support.block_test_case import NIOBlockTestCasefrom nio.common.signal.base import Signalfrom ..your_block import YourBlock

class TestYourBlock(NIOBlockTestCase):

@patch('requests.get')def test_some_feature(self, mock_get):

blk = YourBlock()self.configure_block(blk, {

'int_prop': 23,'obj_prop': {

'p1': 'foo','p2': 'bar'

}})blk.start()blk.process_signals([Signal()])blk.stop()

1.10.2 setUp/tearDown

Just like unittest.TestCase, we support the setUp/tearDown pattern. This is a great place to do any initialization and/orcleanup that will be required across every test. Don’t Repeat Yourself! Be aware, though, that we do some crucialinitialization and finalization in NIOBlockTestCase.setUp/tearDown. If you override either of these methods, makesure you call the method in the parent class at the top of your method.



1.10.3 Helper Methods

• configure_block(block, block_properties) - The process of configuring and initializing blocks manually issomewhat nuanced (and not something we want you to worry about), so we provide this method to configureyour block instance semi-automatically. Just pass the block object itself and a dictionary containing any blockproperties you want to configure (and associated values).

• assert_num_signals_notified(num, block=None) - This method provides access to the total number of signalsnotified over the course of the current test. If block is not None, then you’ll get the number of signals notifiedby that block over its lifetime.

1.10.4 Overridable Methods

• get_test_modules() - By default, NIOBlockTestCase automatically initializes the logging, threading, sched-uler, and security modules. However, you can customize this by overriding this method and returning a list ofstrings corresponding to the particular modules you want to initialize. Your options are as follows: * logging *threading * scheduler * security * communication * persistence * web

• signals_notified(signals, output_id) - This method gets called every time signals are notified in your tests. Ifyou’d like to record something in the test case, trigger an event, or perform some aggregation when that happens,override this method. One common use it to add self.signals = defaultdict(list) to setUp and:

def signals_notified(self, signals, output_id):self.signals[output_id].extend(signals)

1.10.5 Events

This is more a practice than a feature. Blocks are not required to behave synchronously, and sometimes you mightwant to wait for an event (after instantiating or configuring a block) before proceeding with the tests. Rather thansleeping for a prescribed amount of time (asynchronous processes can be fickle and unpredictable from machine tomachine), we recommend extending the block you’re testing and adding one of Python’s Event objects to signifyreadiness. Here’s an example:

class EventBlock(YourBlock):def __init__(self, event):

super().__init__()self.e = event

def configure(self, context):super().configure(context)self.e.set()

Now, instead of instantiating YourBlock in your tests, you should instantiate EventBlock, passing an instance ofEvent to its constructor. Check this out:

e = Event()blk = EventBlock(e)

self.configure_block(blk, {'p1': 23})e.wait(2)

This way, your test will wait until the meat of YourBlock.configure has returned control to the method on the childclass. Your test will never proceed until EventBlock.e is set.

1.10. Block Testing 21


1.10.6 Mocking

Patching and mocking are extremely useful concepts in software verification; this is especially relevant when themodules in question interact with external resources (e.g API’s, OS services, etc.). We won’t go too much into thedetails of mocking right now, but the [Python documentation](https://docs.python.org/3/library/unittest.mock.html)contains a ton of great material on the subject. We recommend that you use these concepts liberally; in fact, we expectthat, in many cases, you won’t have much choice.

As you progress, one thing you might notice is that unittest.mock.patch doesn’t play nice with relative module paths.This can be a pain when you want to patch a method at the class or module level. One solution is to import the objectdirectly and use unittest.mock.patch.object:

from unittest.mock import patch, ANYfrom ..queue_block import Queue

@patch.object(Queue, '_load')def test_it(self, load_patch):

...load_patch.assert_called_once_with(ANY)

Again, you don’t necessarily have to construct your tests in this way; however, we’ve found this practice to be moreconvenient and less prone to user error than others, so we thought we’d pass it along to you.

Mocking Persistence Module

To mock load:

class TestPersistenceBlock(NIOBlockTestCase):

def test_persist_load(self):blk = Block()with path('nio.modules.persistence.default.Persistence.load') as load:

load.return_value = 'i was persisted'self.configure_block(blk, {})

To mock store and save:

from unittest.mock import MagicMock

class TestPersistenceBlock(NIOBlockTestCase):

def test_persist_save(self):blk = Block()self.configure_block(blk, {})blk.persistence.store = MagicMock()blk.persistence.save = MagicMock()

1.10.7 n.io Modules

NIOBlockTestCase configures the following n.io modules by default: [’logging’, ’scheduler’,’security’, ’threading’]. If your block test case needs to use any other n.io modules then you need tospecify that by implementing the get_test_modules method.

For example, if your test case uses persistence:


https://docs.python.org/3/library/unittest.mock.html


class TestBlock(NIOBlockTestCase):

def get_test_modules(self):return super().get_test_modules() + ['persistence']

You can override the default configuration of modules by implementing get_module_config_*. The followingexample will ensure that the test case uses the default implementation of the persistence module.

class TestBlock(NIOBlockTestCase):

def get_module_config_persistence(self):return {'persistence': 'default'}

1.11 Block Router

The block router is what manages the passing of signals from block to block. nio comes packaged with a variety ofblock routers and each is configurable to control behavior.

1.11.1 Configuration

Configuration of block routers is set in the block_router.cfg file in the project’s etc/ directory.

• clone_signals: False

• If clone_signals is True, then the block router will perform a deepcopy on the signals whenever they are splitand passed to more than one block.

• max_workers: 50

• When using ThreadedPoolExecutorRouter, this is the max number of workers.

1.11.2 Block Router Types

Each nio project can specify a default block router in nio.conf by setting block_router.

In addition to specifying the block router for the project, each service can use its own block router by settingblock_router on the service configuration.

BaseBlockRouter

Default block router.

nio.common.block.router.base.BaseBlockRouter

ThreadedBaseBlockRouter

Threaded block router.

nio.common.block.router.threaded.ThreadedBaseBlockRouter

ThreadedPoolExecutorRouter

Threaded block router that makes used of thread pools (https://docs.python.org/3/library/concurrent.futures.html).

nio.common.block.router.thread_pool_executor.ThreadedPoolExecutorRouter

1.11. Block Router 23

https://docs.python.org/3/library/concurrent.futures.html


1.12 Block Context

In the configure method, blocks are passed ‘context’ about themselves and the environment that they are running in.This information is available to be used by block developers:

• block_router (BlockRouter):The router in which the block will be running. The only requirements are that itcan handle signals notified by its blocks.

• properties (dict): The block properties (metadata) that will be deserialized and loaded as block properties.

• hooks (Hooks): Hooks to subscribe to participate in the lifecycle process involving all blocks running together.

• service_name (str): The name of the service this block belongs to.

• command_url (str): The URL at which this block can be commanded. This URL will not havehost or port information, as that may be different based on public/private IP. It will look like “/ser-vices/ServiceName/BlockAlias/”.

• mgmt_signal_handler (method): method to use to publish management signals


Table of ContentsGetting StartedBuilding Your First NIO ApplicationNIO ExpressionsNIO Command Line ToolsNIO log level component APIComponent - Management PublisherREST APICommunication ModuleBlock DevelopmentBlock TestingBlock RouterBlock Context

nio documentation · •python 3.4 or greater •pip •virtualenv •git first, open a terminal in...

Documents