University of Southern Denmark

Online computational ethology based on modern IT infrastructure

Larsen, Leon B.; Neerup, Mathias M.; Hallam, John

Published in:Ecological Informatics

DOI:10.1016/j.ecoinf.2021.101290

Publication date:2021

Document version:Final published version

Document license:CC BY

Citation for pulished version (APA):Larsen, L. B., Neerup, M. M., & Hallam, J. (2021). Online computational ethology based on modern ITinfrastructure. Ecological Informatics, 63, [101290]. https://doi.org/10.1016/j.ecoinf.2021.101290

Go to publication entry in University of Southern Denmark's Research Portal

Terms of useThis work is brought to you by the University of Southern Denmark.Unless otherwise specified it has been shared according to the terms for self-archiving.If no other license is stated, these terms apply:

• You may download this work for personal use only. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying this open access versionIf you believe that this document breaches copyright please contact us providing details and we will investigate your claim.Please direct all enquiries to puresupport@bib.sdu.dk

Download date: 31. May. 2022

Ecological Informatics 63 (2021) 101290

Available online 3 April 20211574-9541/© 2021 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Online computational ethology based on modern IT infrastructure

Leon B. Larsen *, Mathias M. Neerup, John Hallam University of Southern Denmark, Denmark

A R T I C L E I N F O

Keywords: Bioacoustics Recording Automated annotation

A B S T R A C T

In the study of animal behaviour, annotation and analysis is largely done manually either directly in the field or from recordings. An emerging field, computational ethology, is challenging this approach by using machine learning to automate the process. However, the use of such methods in general is complicated by a lack of modularity, leading to high cost and long development times. At the same time, the benefits of implementing a fully automated pipeline are often minuscule. We propose online analysis as a way to gain more from automating the process, such as making it easier to ensure that equipment is properly configured and calibrated, enabling the recording equipment to follow the animals, and even enabling closed-loop experiments. In this work, we discuss the requirements and challenges for such a system and propose an implementation based on modern IT infra-structure. Finally, we demonstrate the system in case studies of bats and mongoose. As more and more methods and algorithms are developed we expect online systems to enable new experimental setups to study behaviour, leading to new insights in the field.

1. Introduction and principles

The behaviour of animals has traditionally been studied by obser-vation using both qualitative and quantitative methods (Tinbergen, 1963). In both cases, scoring or timing of behaviours has been done manually with pen, paper and stop watch (Goodwin et al., 2020; Martin et al., 1993). As technology has advanced, the use of recordings has become much more frequent introducing a two stage process where data is first collected and then, at a later time, annotated and analysed. The quality of recordings is a challenge to ethologists since they prefer to record the animal in its natural environment (Egnor and Branson, 2016) where it cares neither about staying close to the microphone nor in view of the camera. This often leaves the researcher with a trade-off between poor data quality under natural conditions or better data quality under more constrained conditions (Datta et al., 2019). Means to improve the quality of data captured under natural conditions are thus desirable.

Leveraging machine learning, computer vision, and artificial intel-ligence for automated annotation and analysis especially requires high quality data (Egnor and Branson, 2016). Although automated annota-tion is widely applied, it is mostly used offline which has the advantage that the researcher can assist the algorithms, fine tune parameters and oversee that classifications are correct. However, it also limits the op-tions for data collection and closed loop experimentation. We instead propose to automatically annotate and analyse data online while the

recordings are made. This has at least three advantages: 1) It makes it easier to ensure that the system is properly configured and calibrated, because the scientist can inspect the annotated output during experi-mentation, 2) it enables the recording equipment to react to the animals, for example by orienting a sensor or selecting between sensors, and 3) it enables closed-loop experiments, for example controlling stimuli based on the behaviour of the animal. In this paper we discuss the re-quirements and challenges for such a system and propose an imple-mentation based on modern IT infrastructure. We demonstrate the system in two case studies: 1) online call annotation for multi-channel, long term recordings of bats, and 2) online activity detection in multiple- view video recordings of mongoose.

2. State of the art

Technological advances have enabled automated annotation and analysis, known as computational ethology, permitting new types of experiments (Anderson and Perona, 2014). For example, in the study of vinegar flies an automated pipeline enables high-throughput screening of behavioural variation (Werkhoven et al., 2019a, 2019b). Another example is from the study of rats, where advances in computer vision (Mathis et al., 2018; Nath et al., 2019) enable long-term behaviour analysis of freely moving animals (Jain et al., 2020). Also the birdsong community has benefited for some time from automated segmentation

* Corresponding author. E-mail address: lelar@mmmi.sdu.dk (L.B. Larsen).

Contents lists available at ScienceDirect

Ecological Informatics

journal homepage: www.elsevier.com/locate/ecolinf

https://doi.org/10.1016/j.ecoinf.2021.101290 Received 14 January 2021; Received in revised form 25 March 2021; Accepted 25 March 2021

Ecological Informatics 63 (2021) 101290

2

of syllables and measurement of similarity between vocal behaviours (Tchernichovski et al., 2000).

The need for computational ethology methods is increasing as the amount of data collected is far exceeding the capacity for hand anno-tation. This is for example the case in studies of marine mammals using acoustic recording tags (Johnson et al., 2009) producing immense amounts of data for offline processing, a method also moving to other taxa, such as bats (Stidsholt et al., 2019). Similarly, wild life cameras and other long-term recording devices producing data, often with sparse activity, year round require methods for detecting points of interest.

There are good reasons for preferring automated behaviour analysis. It both reduces the amount of tedious labour and it makes the results more consistent and reproducible (Anderson and Perona, 2014; Good-win et al., 2020). In some cases the human sensory system is actually incapable of making the observations, for example in high speed tracking of individuals, and there are limits to how many variables one can observe simultaneously (Anderson and Perona, 2014).

Despite the advantages of computational ethology, most annotation is still done by hand or at least partly by hand (Goodwin et al., 2020). There are many reasons for this, such as lack of training in biology for engineers or in engineering for biologists (Egnor and Branson, 2016), but mostly we are lacking standardisation and convergence of the methods. This means a system must be built specifically for one exper-iment with little reuse in other setups, making full automation too expensive and cumbersome to be worth while.

A subset of computational ethology setups are capable of analysing behaviour online as it is happening. In this work the word online always mean on the fly or more precisely ‘the ability to handle data fast enough to maintain closed loop operation with the process producing them’. An example of online operation is from thermogenetic experiments with flies, where online tracking made it possible to point an IR laser on the free-walking subjects (Bath et al., 2014). In another study, demon-strating place-cell-like neural activity in primates, wireless neurophys-iology combined with online nocturnal tracking and remote-controlled stimulus-foodboxes were used to form a closed-loop interaction with the animals (Nourizonoz et al., 2020). Another example is optogenetic (Fenno et al., 2011) closed-loop experiments where the brain of the animal is stimulated based on simultaneously observed dynamics (Grosenick et al., 2015). In these examples, online analysis of behaviour enabled closed loop experimentation.

It is also common that a human takes the role of online behaviour analyser such as in animal psycho-physics where the animal is trained to station and then react if it picks up the presented stimulus (Anderson Hansen et al., 2020; Larsen et al., 2017; Sørensen et al., 2020). Here the researcher decides when the animal is ready and evaluates the reaction. The approach has also been used in experiments with songbirds, where a researcher detects when the bird is singing and releases a masking noise, making the bird change its pitch (Brumm and Slabbekoorn, 2005; Lohr et al., 2003). Many such applications could benefit from online computational ethology, but require it to be available in a modular form so the researcher only needs to define the logic.

An advantage of online analysis is that problems can be noticed and fixed earlier in the recording process. For example in long term re-cordings, the researcher can follow results as they are coming in. Online analysis also enables the sensors to react to the behaviour of the animals for example if they are moving around. Finally, online analysis enables closed-loop interaction with the animals for example enabling auto-mated experimentation or interactive biorobotics (Datteri, 2020).

There are both open-source and proprietary software solutions that to some degree can support online analysis. Each is an optimisation of certain properties, such as timing requirements for use in online spike sorting (Nguyen et al., 2014), processing of spike data (Ciliberti and Kloosterman, 2017), and tracking of flies (Werkhoven et al., 2019b). Some are optimised for user friendliness, for example through a visual programming language (Lopes et al., 2015). Some are based on specific hardware or methods relevant to the particular experiment, such as

multi-channel closed-loop electrophysiology (Siegle et al., 2017), ul-trasonic recordings of bats (Andreassen et al., 2013) or feature extrac-tion and clustering of bird song (Tchernichovski et al., 2000). Each has its own incompatible architecture and software that cannot easily be combined with another. It is not uncommon that frameworks, archi-tectures and software packages are developed with great potential and intentions, only to have a very short lifetime before no longer being maintained. This can be due to lack of funding, changing requirements or simply because the one person who understood how it works moved on to other challenges. We think the best way to prevent this is to reduce the amount of code that needs to be maintained by using free and open source software and to make sure that any person trained in software development can maintain it.

In summary, we want to take advantage of the rapid development in machine learning without having to constrain the environment too much. We want to enable re-use of developed components in other experimental setups to improve standardisation and convergence of methods. Further, we want to base the solution on mature well-known technologies and off-the-shelf hardware to keep it easy to maintain.

We therefore propose an architecture, based on modern IT infra-structure, that delivers on those requirements. In the following, we describe the architecture and explain why we think the solution should build on IT infrastructure.

2.1. Architectural considerations

The architecture we propose for online computational ethology is built on two fundamental computer science data abstractions: streams and events. A stream is a sequence of data values without natural beginning or end while an event is an association between a collection of properties and a range of time. A natural corollary is that one can have a stream of events, i.e. an event is also a suitable data structure for streaming.

Animal recordings are inherently streams because they have no natural beginning or end and ethology is inherently event-based in that it attributes a specific time or time range with a specific behaviour or set of behaviours (Kim, 1993). For example, audio recordings of a songbird consist of a stream of samples, while the behaviour of interest could be a bout represented by the onset, offset, and sequence of syllables making up the bout.

A key feature to support in the architecture is the processing neces-sary to move between the two domains. This implies inferring events from data streams meaning that the stream must be segmented to know the time and the segment must be classified to know the behaviour (Datta et al., 2019). To generate the events online, segmentation and classification needs to happen fast enough for the application.

Robots inherently operate in an online fashion and thus robotics is a natural field to inspire an online architecture. A common robot archi-tecture (Fig. 1) is a hierarchical layered, closed-loop system (Russell et al., 1995) connecting the robot to the environment through sensors and actuators. Starting from its sensors data passes through different processors performing feature extraction, pose estimation, object recognition etc. A top level program then makes a decision about what actions to take and the actions are synthesised through layers of kine-matics, predictors and planners to the actuators (Jensen et al., 2014). There are many operating systems and middle-wares available imple-menting such an architecture which could help inform the design of an online architecture and each has its strengths and weaknesses. The Robot Operating System (ROS) has been very successful especially in research (Quigley et al., 2009). It is a highly modular system in which code is distributed into individually compiled nodes and combined at run-time. The infrastructure of ROS follows the publish-subscribe pattern (Birman and Joseph, 1987) where nodes can publish messages of a predefined type on a topic for other nodes to subscribe to (Fig. 2). For example the only concern of a camera node is to publish the video stream from the camera and another node processing video data is

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

3

agnostic to where it came from. This infrastructure ensures modularity and scalability because the nodes are independent of each other.

Modular design is an overarching goal for the architecture meaning that it should generalise to different applications and encourage re-use of modules. This goal is also shared with many distributed systems, such as the Internet-of-Things (Nord et al., 2019) and industrial Super-visory Control And Data Acquisition (SCADA) systems (Yadav and Paul, 2020). Here, it is common to associate each sensor with a bit of pro-cessing power, enabling it to stream its data directly on the network. A system for online computational ethology should take advantage of the development in sensors for such systems supporting the goal of using off- the-shelf hardware.

2.2. The proposed architecture

The proposed architecture is shown in Fig. 1 together with the robot architecture from which it was adapted (Jensen et al., 2014). Boxes represent nodes, arrows represent data, and in both cases data is ordered in topics.

Raw data from the environment is immediately published as data streams by the streaming interface. In practice this means that sensors stream data live directly on the network without processing or storing anything. Contrary to the original architecture where all sensors are connected to a central computer this implies that sensors have pro-cessing power.

Event encoding handles segmentation and classification of the behaviour. It is shown at the edge of the cloud because it can reside either inside or outside depending on the application. In some cases the streaming layer can be bypassed if the sensor directly emits events. An example could be a lever that is either on or off.

Ellipses inside the cloud represent event processing services. Those

could, for example directly drive actuators to release a reward or they could combine multiple events in logic relations or state machines to form complex events. An example could be activation of a lever within 500 ms after a sound being played back results in a the-animal-could- hear-the-sound event.

Similar to event encoding, event decoding also resides at the edge of the cloud. In some cases events can directly influence the environment, for example by opening a reward box, and in other cases the event decoder must generate a stream, for example to drive a motor.

The two main abstractions in the architecture are the node and the interface (Fig. 3). A node contains processing power and can be con-nected to hardware such as sensors, actuators or storage devices. An interface is a network-streaming entity connecting a node to the network. From these two abstractions, three main node types can be constructed (Fig. 3): publisher nodes, subscriber nodes and processor nodes. Each node can run on a separate computer, multiple nodes can run on the same computer or a node can run in a container.

The last key abstraction is the topic. A topic is a specific stream to which a node can publish or subscribe. Thus the topic is the mechanism that allows nodes to receive only part of the data present on the network.

Because the architecture is data-centred, the publisher node does not care if anyone subscribes to the data it produces and similarly the sub-scriber node has no knowledge of where the data it consumes comes from. There can be multiple publishers to a topic, a publisher can publish to multiple topics and a node can subscribe to multiple topics (Fig. 2).

2.3. Implementation considerations

One way of achieving timely operation is by parallelisation, meaning that the task is split between multiple processing units. For example the classification of a segment may be independent of classifications of other segments so multiple segments can be processed simultaneously. Depending on the requirements, there are different ways to scale this.

Fig. 1. Left: The classic architecture for a robot (Russell et al., 1995) adapted from (Jensen et al., 2014). Right: The proposed streaming based architecture.

Fig. 2. The publish-subscribe pattern where producers can publish data, sub-scribers can consume data and processor nodes can do both (Birman and Jo-seph, 1987).

Fig. 3. Left: Shows the main abstractions of the architecture: the interface and the node. The manager is only present in the system when needed. Right: The three node types.

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

4

Hard real-time constraints suggest keeping data in the memory of one machine having many local processing cores. This method, where more work is achieved by adding resources to the computer, is known as vertical scaling and has obvious limits.

The alternative is to scale horizontally by adding more machines connected in a network. This introduces a delay because data must propagate over the network but greatly improves scalability of the so-lution. In practice, horizontal scaling is often achieved using containers. Seen from the program running in the container it is indistinguishable from just running on a normal computer but in fact many containers can share the resources of a computer or a cluster. The online system could benefit from containerisation to relieve the programmer from the re-sponsibility for advanced parallel programming and to ensure that programs are self-contained, thus generalising to different applications.

Streams are very common in modern IT where they are used for carrying video and sound in multimedia services like Netflix, Spotify and Zoom. The principle is that a small chunk of samples is compressed, wrapped in a network packet and sent via the internet. The underlying protocols and infrastructures are very mature, integrated and easy to use. An online system could benefit from the timely operation of such mature software and the fact that many developers know how it works.

An event consists of a timestamp and a payload. The payload is expressed in key-value pairs containing integers, floats, strings, arrays or objects and it can be nested to contain for example an array of objects. Like data streams, event streams are organised in topics following the publish-subscribe pattern and can share the infrastructure of the data stream layer.

Current technologies for event-stream processing have matured as the backbone for social media, e-commerce and financial markets. The principle is that a number of message brokers receive events and maintain a list of where to send them. Clever implementation makes it possible to guarantee that each message is delivered exactly once to each recipient. Such systems scale extremely well and can deliver events in close to real-time.

As event-stream methods have gained more popularity, algorithms and methods are being adapted to online processing. Online detection of novelties (sometimes called anomalies or outliers) has received consid-erable attention for instance in security systems to discover the behav-iour of a hacker trying to penetrate the system (Ding et al., 2016; Lodi et al., 2014; Nisha and Pramod, 2019) or in predictive maintenance to replace parts before they break (Naskos et al., 2019; Pawlowski et al., 2014). Such algorithms come in both unsupervised and model-based flavours (Angelov, 2013). Also machine learning methods for clus-tering (Angelov, 2004; Baruah and Angelov, 2012; Bharitkar and Filev, 2001; Filev et al., 2010) and classification (Hoi et al., 2013; Ying and Zhou, 2006) are available in online versions. Good methods for easily visualising events have also been developed and are very mature. A system for online computational ethology should take advantage of event-processing to leverage the huge investments made in these technologies.

The architecture is master-less meaning that each node will continue with its current settings indefinitely. To manage the system, for example changing parameters, nodes are re-deployed with the new settings. This is similar to how web services are typically connected. For example a web server would have a well-known address to contact a specific database on and it would have fixed parameters for what to do with the data. To keep such systems consistent, provisioning software is used to deploy the nodes based on a central repository of settings. This way, all the parameters that are hard-coded in the nodes are actually controlled from central configuration files.

Although the goal is an online processing architecture, it also needs to support offline analysis. In SCADA systems, among others, this is achieved with a historian; a universal subscriber/publisher responsible for saving and replaying raw data from the data streams. It is essentially a processing node connected to a storage device, but it is recommended to have a designated computer for this task close to and on the same

physical network as the other nodes in order to prevent data loss caused by connection failure. Upon request a historian node re-streams captured data back into the system either directly or after rewriting network headers, for example to assign data to another topic. This means that all other nodes are agnostic to whether data is live or playback. Data saved by the historian can also be exported for further offline processing. Because packets are time stamped on arrival, synchrony is retained under the assumption that latency is the same for all packets which could be violated on a congested network. Time stamping can also be done by the producer using the Real-Time Protocol (Schulzrinne et al., 2003) if need be. There is a limit to how much traffic can be captured by one historian but multiple historians can be deployed capturing different multi-cast groups.

3. Example implementation

The example implementation has the data stream layer implemented using the latest standard for the internet, IPv6 (Deering and Hinden, 2017). IPv6 offers stateless address autoconfiguration and thus elimi-nates the need for a designated machine to manage network addresses (DHCP server). IPv6 also features multi-cast which directly implements the topic abstraction because a node can publish data to a specific address constituting a group and the network infrastructure will deliver the data to any subscriber that has joined the group (Costa and Vida, 2004; Deering and Conta, 1998; Holbrook et al., 2006).

An important detail in multi-cast is that copying packets to several subscribers is handled by the network hardware (routers and switches) instead of by the publishing node thus reducing the demands on a node and retaining scalability. It is required that routers and switches know which hosts on the network are in specific groups (they must implement MLD snooping) and that they can inspect the contents of packages to determine where to send them (layer 3 operation). Without this all packages will be sent to all hosts which will work for smaller networks but impacts scalability.

The streaming interface is implemented in the open source cross- platform gstreamer framework (Gstreamer, 2001) and is responsible for joining the respective groups, for encoding and decoding the data, sending it on the network. The data is packaged in UDP datagrams (User Datagram Protocol (No. 768), 1980) as this is optimised for streaming frequent, small payloads and reduces overhead to a minimum. It also means that the receiver has no way of recognising packets that arrive out of order. The standard solution is to pre-package data using the Real- time Transport Protocol (Schulzrinne et al., 2003) providing facilities like jitter compensation, packet loss detection and packet ordering. Both are standard solutions used in multimedia streaming. If data needs to be timestamped by the publisher, the Real-Time Control Protocol (Hui-tema, 2003) also supports sub-microsecond synchronisation.

All nodes have processing power and are able to connect directly to the network. This means that a sensor connected to a Raspberry Pi is sufficient, but an eight channel recording array interfaced to a PC could also stream data to the network. All that is needed to turn any sensor system into a publisher node is a piece of software allowing data to be streamed, i.e. a streaming interface.

A historian is implemented by capturing traffic from the network and directly saving it to disk in files of 1 min duration. This ensures mini-mum processing overhead at the time of capture since no conversion has to be made. The network traffic is saved in pcap files (Duarte and Far-ruca, 2010), a format invented exactly for the purpose of capturing network traffic. Data can be replayed by the historian, streaming them back to the network with the same timing as when they were captured. This is very practical for testing new nodes. Data can also easily be extracted from the pcap files for offline processing. In this implementa-tion there is only one historian subscribing to all topics.

The manager is a laptop only present on the network when deploying nodes. The manager deploys and re-deploys nodes via Secure Shell (Ylonen, 1996) using ansible (Ansible, 2021.) for mass software

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

5

provisioning. The event encoder, event decoder and event processors (Fig. 2.1) are

implemented as container images in docker (Docker, 2013) and run in a cluster or on dedicated PCs. Event processors are thus containers taking data and/or event stream input and producing event stream output. A multitude of event-stream processing tools and systems exist both commercial (Apache, 2011), open source (Vohra, 2016) and aca-demic (Appelrath et al., 2012; Barga et al., 2006; Cugola and Margara, 2012). Event streams, in our implementation, are handled by the open source distributed stream-processing software platform Kafka (Vohra, 2016). Kafka runs on multiple servers comprising a Kafka cluster and events published on topics are stored in the cluster. Kafka handles load balancing through its partition system and distribution ensures high availability. Kafka also handles all copying and distribution of the messages ensuring that each message is delivered exactly once to each subscriber.

In the case studies presented here, streams are transformed into event streams by estimating a feature in the data stream and inferring the events from that. For example, we calculate the power in an audio signal and generate an event when it exceeds a threshold. In practice, we use two thresholds in a principle called hysteresis (Fig. 4), to prevent situations where the signal fluctuates around the threashold, generating multiple events. In the case of an active-high value being converted to events, an onset event is emitted when the value becomes greater than the upwards threshold tu. When the value becomes lower than the downwards threshold td an offset event is emitted.

The inverse transformation from events to value is made with an adaption of the kernel principle (Parzen, 1962; Rosenblatt, 1956). An onset event changes the internal state to count upwards and an offset to count downwards. The output value is simply the value of the counter representing the activity of the input event. Fig. 4 shows the count with linear unit gain in both directions, but any function could be used and it could have different parameters for counting up or down. This way different patterns can be weighted to form different measures, for example letting many short bursts give a higher output than fewer long

bursts. Assuming that sensible thresholds are chosen for generating the events, the activity estimate from different input sources can be com-bined to infer complex events from multiple sources. For example, if both power and entropy events have been emitted, it means that a loud sound, unlikely to be noise, was picked up and this could trigger a vocalisation event.

A web interface (Neerup and Larsen, 2019) provides the user with random access to live data or recordings played back from the historian. The interface is configurable by adding, removing, organising or resizing panes. When adding a pane, a drop down menu allows the user to choose between available streams. The user can also add dashboards, for example to visualise events, and by clicking on the dashboard select interesting time periods. The backend automatically deploys the necessary containers and preserves state coherence between client and server. The user can also deploy plugins, for example to visualise audio data. The interface builds on modern and commonly used frameworks such as docker (Docker, 2013), React (Occhino, 2015), janus (Amirante et al., 2014), Elasticsearch (Elastic, 2012) and Kibana (Elastic, 2016) making it easy to extend, adapt and maintain.

Synchronisation between nodes is handled by chrony (Chrony, 2021), an implementation of the NTP protocol (J. Martin et al., 2010). It keeps the nodes on the network synchronised to an accuracy of tens of microseconds, which is sufficient for the applications presented here and sub-microsecond accuracy is possible. This eliminates the need for timestamping in the data stream layer and since data is always processed as fast as possible the policy is to skip unprocessed data if newer data is available (if this policy is unacceptable, the use of TCP or RTP can ensure lossless communication at the price of increased and variable delay). The historian timestamps data when it is received and on the event stream layer timestamping is naturally done by the publisher since the timestamp is part of the event.

4. Case: multi-channel recordings of wild bats

An eight-channel ultrasonic recording system (Andreassen et al., 2013) was deployed on a rooftop next to a forest (Fig. 5A). The purpose was long-term recording of bats to study their activity patterns and the prevalence of different species. For this work, however, it serves as a demonstration case showing that online detection of onset and offset events in a real outdoor setting is possible. Applications for this could be to provide feedback to the researcher while setting up the system, to quickly identify interest points in newly produced data or to have the microphone array align automatically towards the sound.

The data stream layer consists of one publisher node, a historian node (to save the raw data) and an event emitter to estimate the power contents of the sound signal and emit onset/offset events. Since the microphone array already has processing power, the publisher node could be deployed directly on that. The eight data streams produced were published on a common topic in order to keep the timing tight enough for later use in localisation, but for this work only one channel was used. The historian was deployed on a local machine close to the setup in order to buffer the recorded data and thus prevent data loss in case of connection fall-outs. The event emitter was deployed in a docker swarm that for this case could have been limited to one computer.

The elements developed for this application are 1) a streaming interface for the microphone array, 2) an event-emitter running in the cloud and estimating the power in one audio channel, 3) a plugin for generating online spectrogram video, and 4) a Kibana visualisation dashboard to visualise events.

Fig. 5C shows three plots from a 10 s recording to demonstrate the timing of the system. The top plot shows the estimated power value and the bottom plot shows the spectrogram. In the centre plot the events are shown with onsets as positive stems and offsets as negative. Manually counting the calls yields 56 while 48 were detected. Closer inspection shows that 14 calls were not detected while 11 calls were double detected. 36 calls out of 45 were correctly detected corresponding to

Fig. 4. Top left: Shows the conversion from a value (black curve) to events. Onset events are shown as positive blue stems and happen when the value crosses the upwards threshold denoted tu. Offset events are shown as negative red stems and happen when the value crosses the downwards threshold td. Top right: Shows the same principle for an active low signal like entropy. Bottom: Shows the conversion from events to an estimated value based on an online adaptation of the kernel principle. In the period between online and offline events the value (black line) is increasing and in the period between offline and online it is decreasing. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

6

80%. Some of the calls are very hard to tell apart in the spectrogram and contain very little energy so tuning the system to annotate those is likely to produce more false positives. Even if detection is not perfect, it is sufficient to indicate that bats have been active in the area or not and thus support a decision about moving the array.

The delay through the system consists of several parts. First there is a processing delay on the Raspberry Pi and from latency in the network. That time was measured to be 306 ms and less than 10 ms stems from the network. The power estimates are made at 320 Hz meaning that the maximum delay in that part is 6 ms (twice the period) and finally there is a small unknown delay from the event being published until it is received by a subscriber. A conservative guess about the total delay is half a second thus making it plausible, for example to use the events to trigger an artificial stimulus. The recording had a reasonably high signal to noise ratio (~ 40 dB) making it possible to apply the threshold- crossing method for event generation. If deployed in a real application more elaborate methods with adaptive thresholds might be needed. The recording also had a very high sampling rate (500 kHz) which for eight channels means streaming more than 64Mbit/s. Since the system was implemented with gigabit hardware this did not cause problems. The application also demonstrated the principle of event generation reducing the amount of information from the 10.6MiB per channel originally streamed (in ~ 10 s) into 96 events (<4KiB). Since each channel is handled by a container it seamlessly scales to multiple channels. Finally the case serves as a demonstration of the idea that any sensor, in this case a multi-channel ultrasonic recording array, can be adapted to the online architecture.

5. Case: multi-camera recordings of banded mongoose in a zoo

A six-camera system was deployed in a mongoose enclosure at

Odense Zoo (Fig. 6A). The purpose of the project was to provide video recordings for a study of the effects of environmental enrichment in a group of captive banded mongooses (Mungos mungo). For this work, however, it serves as a demonstration case to show that camera feeds can be processed online using a Convolutional Neural Network (LeCun et al., 2015). The application was a monitor for the zoo guests which always shows the camera feed containing most animals. An alternative appli-cation could be measuring the spatial preferences of the animals over time.

A camera node based on a Raspberry Pi was developed (Fig. 6C) so that it could be placed safely in the animal enclosure, and so that it could withstand cleaning with a pressure washer. The setup consisted of six cameras placed in and around the enclosure (Fig. 6B) and a PC running the historian node placed in the basement under the enclosure. Since the six video feeds were collected for another project and the application presented here was developed while the experiments were running, the monitor system was never deployed in the zoo but was instead tested in the lab using data streams recorded at the zoo. However, online pro-cessing of the re-streamed data is fully equivalent to processing live data, hence the demonstration is still valid. The video feeds from the indoor part of the enclosure suffered from changing light conditions making it hard to detect the animals. Since they rarely used their enclosure within opening hours, it was decided to only use the three outdoor cameras (Fig. 6D).

The elements developed for this application are 1) a camera node, 2) a tensorflow-based (Abadi et al., 2015) event encoder running on a PC with NVIDIA 1080 graphics card, 3) a selector node that emits state change events when the video feeding the monitor should change, 4) a display node connected to a monitor and switching between input video streams based on events, and 5) an annotated training set containing 500 images with 4 classes (mongoose, bird, guest and trolley) used for

Fig. 5. A: The microphone array deployed on the roof of a building next to a forest. B: The multi-channel ultrasonic recording system hardware consisting of a Raspbarry Pi, a USB-DUX and an in-house interface PCB. C: Three plots superimposed to show the timing of events. The bottom plot shows the spectrogram from one channel of a 10 s recording of the brown bat (Nyctalus noctula). The top plot shows the sum of columns in the spectrogram as an estimate of the power contents. Since the microphone was not calibrated the units are undetermined. The center plot shows onset and offset events obtained from thresholding the power estimate at tu =

0.25 and td = 0.1. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

7

training the tensorflow model (200,000 steps) on the Abacus super computer (Abacus hpc, 2021).

In Fig. 6E the state transitions of the mongoose video feeds are visualised. Each line shows the number of mongooses detected in the given camera feed at that time. The bottom row shows the output from the selector node marking which of the camera feeds were shown on the monitor at that time. A low pass filter was applied in the selector node to prevent flickering between two cameras. In Fig. 6F a frame from camera 3 is shown with a visualisation of the bounding boxes from the classi-fication. This ability to draw bounding boxes is a feature of the display node which is useful for debugging and testing but is not used in the application.

We were able to detect mongoose online in a video stream at a rate between 10 and 15 frames per second. The events were used to control a monitor showing the video stream with most mongoose in it. The delay through the data stream layer was measured as the time from a light turned on in the camera view until that light was displayed on a display node subscribing to the topic. This was measured to 391 ms which is a conservative estimate since it includes the delay in the monitor node. The classification delay varies a lot but a best case value would be twice

the period at 15 FPS which is 133 ms and it was measured to 234 ms on average. Worst case is that it is never detects the animal for example when they are resting very close together or when otherwise occluded. A conservative estimate is a total delay less that 1 s which is fine for the monitor application but could cause problems in other setups. For ap-plications like position preference or activity measurements the accu-racy of the detector is more important than a small delay. Most of the time the system was able to correctly classify all the mongoose present in the frame, but as with any deep learning approach the detection relies heavily on the amount of training data and 500 labelled frames is still a very small set. When trained properly it is normal to achieve accuracy around 95% (Kovalev et al., 2016). It would clearly be possible to optimise this application by providing more labelled data or use faster hardware, but maybe also by adding functionality to track each animal since their possible movement between frames is limited.

6. Discussion

The proposed architecture can be used in both descriptive and experimental research designs. Studying wild bats is an example of a

Fig. 6. A: Map of the enclosure showing the camera positions. B: The field of view from the six cameras (top row cam 2, 3, 5 and bottom row cam 4, 6 5). C: The camera adjustment system capable of adjusting pan and tilt of the camera from outside the box. D: Examples of the mounted cameras (3,6 and 2). E: The number of animals detected per camera and at the bottom the output from the selector node. Only three cameras are shown since no mongoose were present in the view of the remaining three cameras. F: Example frame from the video stream superimposed with bounding boxes of the animals found.

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

8

naturalistic observation (descriptive design) while the mongoose case is a field experiment (experimental design). It could also be applied in more controlled lab experiments where off-site computation, for example in the cloud, is needed. It is not well suited for applications with hard real-time constraints meaning systems in which deadlines cannot be missed.

There are two factors potentially limiting the scalability of a system, namely contention and cross-talk (Schwartz, 2017). Because the nodes are independent in this system neither of the liming factors are present and thus the system, theoretically, scales perfectly. In other words, it is possible to keep expanding the network to accommodate more nodes and thus the practical limits are way beyond any application we can think of (i.e. running out of IP addresses). It is important to stress that although the architecture scales horizontally, that might not be the case for all applications. For example, not all algorithms support parallelism, which can cause problems with limited bandwidth, especially for the data streams. The event stream implementation also scales horizontally and in both cases the system will scale to any practical experimental setup we can think of. We used on-premise hardware but several Cloud Computing Providers put giant computer clusters at our disposal. This is likely a cheaper and easier solution for many projects compared to establishing and running a cluster.

The tools and methods available for event-stream processing are not the focus of this paper but they are increasingly many. Most have been developed for e-commerce, monitoring of financial markets and IT system monitoring but could as well be applied in biology. AWS kinesis (Amazon, 2011) and Apache Kibana (Elastic, 2016) give access to state of the art machine learning methods for event-stream processing at a click and mature querying languages, such as elasticsearch (Elastic, 2012) make advanced data mining available to a much broader group of users.

Combining many different frameworks, APIs, methods and technol-ogies in one project meant that much of the time normally spent on writing software was instead spent on learning to use existing software. However, once mastered, the use of open free and open source software reduces development time and makes the application easier to maintain. Off-the-shelf components like Raspberry Pi, PiCamera, routers, switches, PCs and graphics cards were used to build the system. The microphone array was the only non-commodity hardware, demonstrating that most sensors can be adapted to streaming. Most biologists would need tech-nical assistance to establish the system and for support. The imple-mentation presented here was built by lab technicians, trained as engineers, in collaboration with engineering students and researchers.

A strength of this architecture is exploiting open source resources, such as gstreamer, tensorflow, ansible, docker, docker swarm, Kafka, Elastic Search, Kibana, Janus etc. Alternatively it could be seen as a vulnerability since updates will be frequent and if one is discontinued it will break our system. It is therefore important to note that they are all commonly used and unlikely to go away in the near future. Also, the use of free and open source software makes it likely that a substitution can be found. The update problem can be handled by locking to a specific version and then doing controlled updates at a suitable time, for example in between experiments. Implementation in docker containers further ensures that each node is self-contained and no updates are needed.

There are many advantages of online processing. It is possible to follow data acquisition more closely, which we found especially useful during long term recordings and in the calibration phase. It is also possible to react to the data being recorded and, for example, auto-matically re-orient the sensors better to capture the focal animal, or to introduce artificial stimuli at biologically significant times instead of fixed time slots and thus interact more meaningfully with the animal.

On the negative side, online analysis may not be possible for all al-gorithms and thus can force the researcher to select a simpler method of detection. It is also more complicated because it will often require custom implementation of methods that are readily available for offline

analysis. However, the two are not mutually exclusive and often has different purposes. For example, counting the bat calls online to help the researcher select a location with more activity would often be combined with more advanced offline analysis to obtain scientific results.

The biggest challenge when deploying hardware and complicated algorithms in a natural setting is often the simplest things. Finding a dry spot for the computers, supplying electricity and establishing a reliable internet connection are examples of this. In our experience it is impor-tant to get out there early in the process to discover the challenges and to put protocols and methods to test. It has been extremely useful in the development of online capable software to be able to record a bag of data and replay it as if it were live data. This was achieved by using pcap files to store network traffic in the historian and replaying that network traffic. This means there are no differences between handling live or replayed data and thus we could for example test online processing of data from the zoo without being there.

A topic that we did not discuss in this paper is edge computing (Davis et al., 2004; Shi et al., 2016), essentially meaning to process data before sending it on the network and thus improve response times and save bandwidth. This requires processing power close to the sensor, which we have, but we used it to stream data to the network and not to reduce the amount of data to transmit. However, a reduction does happen at the edge of the cluster where data streams are transformed to event streams and could thus also happen at the sensor, given sufficient processing power for the task. An important advantage of reducing the amount of data is that data can be transferred from remote locations via low bandwidth solutions, such as LoRa (a long range, low-power, wide area network modulation technique) or Starlink (satellite based internet ac-cess). Although the architecture and online computational ethology in general could support edge computing, this was not the focus of this study.

7. Summary

In this paper, we presented an architecture for online computational ethology that is modular, scalable and easy to maintain. We described an implementation of the architecture based on open source software, internet infrastructure and cloud technology. The goal was to take advantage of the developments in machine learning without having to constrain the environment too much. We have demonstrated this in recordings of wild bats and banded mongoose in an enclosure. As more and more methods and algorithms are developed we expect online systems to enable new experimental setups leading to new insights.

Declaration of Competing Interest

None.

Acknowledgements

We want to thank Thor Andreasen for letting us use one of his recording arrays for the experiments, Sara Sofie Thagaard Winther and Odense Zoo for testing our system, Cao Danh Do for manufacturing the camera boxes, Mads Loose Holst and Martin Krusborg Andersen for training the CNN, Iris Adam and Bridget Hallam for reviewing the paper, Coen Elemeans and Michiel Vellema for sharing their biology experi-ence, Centre for BioRobotics for funding the project and our colleagues in SDU-Biorobotics for discussions and support.

References

Abacus hpc. https://escience.sdu.dk/, 2021. Accessed: 2020-11-01. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Zheng, X., 2015.

TensorFlow: Large-scale machine learning on heterogeneous systems. Retrieved from. https://www.tensorflow.org/. Software available from. tensorflow.org.

Amazon, 2011. Amazon Kinesis. https://aws.amazon.com/kinesis/. Accessed: 2018-08-28.

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

9

Amirante, A., Castaldi, T., Miniero, L., Romano, S.P., 2014. Janus: a general purpose webrtc gateway. In: Proceedings of the Conference on Principles, Systems and Applications of IP Telecommunications. ACM, New York, NY, USA, pp. 7:1–7:8. https://doi.org/10.1145/2670386.2670389. Retrieved from http://doi.acm.org/ 10.1145/2670386.2670389.

Anderson Hansen, K., Hernandez, A., Mooney, T.A., Rasmussen, M.H., Sørensen, K., Wahlberg, M., 2020. The common murre (Uria aalge), an Auk seabird, reacts to underwater sound. J. Acoust. Soc. Am. 147 (6), 4069–4074.

Anderson, D.J., Perona, P., 2014. Toward a science of computational ethology. Neuron 84 (1), 18–31.

Andreassen, T., Surlykke, A., Hallam, J., Brandt, D., 2013, Jun. Ultrasonic recording system without intrinsic limits. J. Acoust. Soc. Am. 133 (6), 4008–4018. Retrieved from.

Angelov, P., 2004. An approach for fuzzy rule-base adaptation using online clustering. Int. J. Approx. Reason. 35 (3), 275–289.

Angelov, P., 2013. Autonomous Learning Systems. Wiley Online Library. Ansible. http://www.ansible.com, 2021. Accessed: 2020-11-01. Apache, 2011. Amazon Firehose. https://aws.amazon.com/firehose/. Accessed:

2018-08-28. Appelrath, H., Geesen, D., Grawunder, M., Michelsen, T., Nicklas, D., et al., 2012.

Odysseus: a highly customizable framework for creating efficient event stream management systems. In: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, pp. 367–368.

Barga, R.S., Goldstein, J., Ali, M., Hong, M., 2006. Consistent Streaming Through Time: A Vision for Event Stream Processing. Retrieved from. http://arxiv. org/abs/cs/0612115.

Baruah, R.D., Angelov, P., 2012. Evolving local means method for clustering of streaming data. In: 2012 IEEE International Conference on Fuzzy Systems, pp. 1–8.

Bath, D.E., Stowers, J.R., Hormann, D., Poehlmann, A., Dickson, B.J., Straw, A.D., 2014. Flymad: rapid thermogenetic control of neuronal activity in freely walking drosophila. Nat. Methods 11 (7), 756–762.

Bharitkar, S., Filev, D., 2001. An online learning vector quantization algorithm. In: Proceedings of the Sixth International Symposium on Signal Processing and its Applications (cat. no. 01ex467), Vol. 2, pp. 394–397.

Birman, K., Joseph, T., 1987. Exploiting virtual synchrony in distributed systems. ACM SIGOPS Oper. Syst. Rev. 21 (5), 123–138. https://doi.org/10.1145/37499.37515.

Brumm, H., Slabbekoorn, H., 2005. Acoustic communication in noise. Adv. Study Behav. 35, 151–209.

Chrony. https://chrony.tuxfamily.org/, 2021. Accessed: 2020-11-01. Ciliberti, D., Kloosterman, F., 2017. Falcon: a highly flexible opensource software for

closed-loop neuroscience. J. Neural Eng. 14 (4), 045004. Costa, L., Vida, R., . Multicast Listener Discovery Version 2 (MLDv2) for IPv6 (No. 3810).

RFC 3810. RFC Editor. Retrieved from. https://rfc-editor. org/rfc/rfc3810.txt. https://doi.org/10.17487/RFC3810.

Cugola, G., Margara, A., 2012, Aug. Complex event processing with T-REX. J. Syst. Softw. 85 (8), 1709–1728. Retrieved from.

Datta, S.R., Anderson, D.J., Branson, K., Perona, P., Leifer, A., 2019. Computational neuroethology: a call to action. Neuron 104 (1), 11–24.

Datteri, E., 2020. Interactive biorobotics. Synthese 1–19. Davis, A., Parikh, J., Weihl, W.E., 2004. Edgecomputing: extending enterprise

applications to the edge of the internet. In: Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters, pp. 180–187.

Deering, D.S.E., Conta, A.. Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification (No. 2463). RFC 2463. RFC Editor. Retrieved from. https://rfc-editor.org/rfc/rfc2463.txt.

Deering, D.S.E., Hinden, B.. Internet Protocol, Version 6 (IPv6) Specification (No. 8200). RFC 8200. RFC Editor. Retrieved from. https://rfc-editor. org/rfc/rfc8200.txt.

Ding, D., Wang, Z., Wei, G., Alsaadi, F.E., 2016. Event-based security control for discrete- time stochastic systems. IET Control Theory Appl. 10 (15), 1808–1815.

Docker, 2013. Docker. Retrieved from. https://www.docker.com/. Duarte, V., Farruca, N., 2010. Using libpcap for monitoring distributed applications. In:

2010 International Conference on High Performance Computing & Simulation, pp. 92–97.

Egnor, S.R., Branson, K., 2016. Computational analysis of behavior. Annu. Rev. Neurosci. 39, 217–236.

Elastic, 2012. Elastic Search. https://www.elastic.co/. Accessed: 2018-08-28. Elastic, 2016. Kibana. https://www.elastic.co/products/kibana. Accessed:

2018-08-28. Fenno, L., Yizhar, O., Deisseroth, K., 2011. The development and application of

optogenetics. Annu. Rev. Neurosci. 34. Filev, D., Georgieva, O., Angelov, P., Kasabov, N., 2010. An extended version of the

Gustafson-Kessel algorithm for evolving data stream clustering. In: Evolving Intelligent Systems: Methodology and Applications, pp. 273–300.

Goodwin, N.L., Nilsson, S.R., Golden, S.A., 2020. Rage against the machine: advancing the study of aggression ethology via machine learning. Psychopharmacology 1–20.

Grosenick, L., Marshel, J.H., Deisseroth, K., 2015. Closed-loop and activity-guided optogenetic control. Neuron 86 (1), 106–139.

Gstreamer, 2001. GStreamer. https://gstreamer.freedesktop.org/. Accessed: 2018-08-28.

Hoi, S.C.H., Jin, R., Zhao, P., Yang, T., 2013. Online multiple kernel classification. Mach. Learn. 90 (2), 289–316.

Holbrook, H., Systems, S., Haberman, B., . Using Internet Group Management Protocol Version 3 (IGMPv3) and Multicast Listener Discovery Protocol Version 2 (MLDv2) for Source-Specific Multicast (No. 4604). RFC 4604. RFC Editor. Retrieved from.

https://rfc-editor.org/rfc/rfc4604.txt. https://doi.org/10.17487/ RFC4604.

Huitema, C.. Real Time Control Protocol (RTCP) Attribute in Session Description Protocol (SDP) (no. 3605). RFC 3605. RFC Editor. Retrieved from. https://rfc-editor.org/rfc/rfc3605.txt.

Jain, K., Menichini, E., Muzzu, T., Macke, J., Saleem, A., Berman, G., 2020. Long timescale dynamics in freely behaving rats. Bull. Am. Phys. Soc. 65.

Jensen, K., Larsen, M., Nielsen, S., Larsen, L., Olsen, K., Jørgensen, R., 2014, Jun. Towards an open software platform for field robots in precision agriculture. Robotics 3 (2), 207–234. Retrieved from.

Johnson, M., de Soto, N.A., Madsen, P.T., 2009. Studying the behavior and sensory ecology of marine mammals using acoustic recording tags: a review. Mar. Ecol. Prog. Ser. 395, 55–73.

Kim, J., 1993. Supervenience and Mind: Selected Philosophical Essays. Retrieved from http://www.loc.gov/catdir/samples/cam034/93000361.html{%}5Cnhttp://www. loc.gov/catdir/description/cam025/93000361.html. https://doi.org/10.1017/ CBO9780511625220.

Kovalev, V., Kalinovsky, A., Kovalev, S., 2016. Deep Learning with Theano, Torch, Caffe, Tensorflow, and deeplearning4j: Which One Is the Best in Speed and Accuracy?.

Larsen, O., Christensen-Dalsgaard, J., Maxwell, A., Hansen, K., Wahlberg, M., 2017, June 9. Cormorant audiograms under water and in air. Acoust. Soc. Am. J. 141 (5), 3667.

LeCun, Y., Bengio, Y., Hinton, G., 2015. Deep learning. Nature 521 (7553), 436–444. Lodi, G., Aniello, L., Di Luna, G.A., Baldoni, R., 2014. An event-based platform for

collaborative threats detection and monitoring. Inf. Syst. 39, 175–195. Lohr, B., Wright, T.F., Dooling, R.J., 2003, Apr. Detection and discrimination of natural

calls in masking noise by birds: estimating the active space of a signal. Anim. Behav. 65 (4), 763–777. Retrieved from.

Lopes, G., Bonacchi, N., Frazao, J., Neto, J.P., Atallah, B.V., Soares, S., Kampff, A., 2015. Bonsai: an event-based framework for processing and controlling data streams. Front. Neuroinformatics 9, 7.

Martin, P., Bateson, P.P.G., Bateson, P., 1993. Measuring Behaviour: An Introductory Guide. Cambridge University Press.

Martin, J., Burbank, J., Kasch, W., Mills, P.D.L.. Network Time Protocol Version 4: Protocol and Algorithms Specification (No. 5905). RFC 5905. RFC Editor. Retrieved from. https://rfc-editor.org/rfc/rfc5905.txt.

Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M., 2018. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21 (9), 1281–1289.

Naskos, A., Kougka, G., Toliopoulos, T., Gounaris, A., Vamvalis, C., Caljouw, D., 2019. Event-based predictive maintenance on top of sensor data in a real industry 4.0 case study. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 345–356.

Nath, T., Mathis, A., Chen, A.C., Patel, A., Bethge, M., Mathis, M.W., 2019. Using deeplabcut for 3d markerless pose estimation across species and behaviors. Nat. Protoc. 14 (7), 2152–2176.

Neerup, M.M., Larsen, L.B., 2019. Online feature extraction and event generation for computer-animal interaction. In: Dockercon19.

Nguyen, T.K.T., Navratilova, Z., Cabral, H., Wang, L., Gielen, G., Battaglia, F.P., Bartic, C., 2014. Closed-loop optical neural stimulation based on a 32-channel low- noise recording system with online spike sorting. J. Neural Eng. 11 (4), 046005.

Nisha, T., Pramod, D., 2019. Sequential pattern analysis for event-based intrusion detection. Int. J. Inf. Comput. Secur. 11 (4–5), 476–492.

Nord, J.H., Koohang, A., Paliszkiewicz, J., 2019. The internet of things: review and theoretical framework. Expert Syst. Appl. 133, 97–108.

Nourizonoz, A., Zimmermann, R., Ho, C.L.A., Pellat, S., Ormen, Y., Prevost-Solie, C., Huber, D., 2020. Etholoop: automated closed-loop neuroethology in naturalistic environments. Nat. Methods 1–8.

Occhino, T., 2015. React. js conf 2015 Keynote-Introducing React Native (Facebook Developers-YouTube).

Parzen, E., 1962. On estimation of a probability density function and mode. Ann. Math. Stat. 33 (3), 1065–1076.

Pawlowski, A., Fernandez, I., Guzman, J.L., Berenguel, M., Acien, F.G., Normey-Rico, J. E., 2014. Event-based predictive control of ph in tubular photobioreactors. Comput. Chem. Eng. 65, 28–39.

Quigley, M., Conley, K., Gerkey, B., FAust, J., Foote, T., Leibs, J., Mg, A., 2009. ROS: an open-source robot operating system. In: ICRA, Vol. 3 (Figure 1), 5. Retrieved from.

Rosenblatt, M., 1956. Remarks on some nonparametric estimates of a density function. Ann. Math. Stat. 832–837.

Russell, S., Norvig, P., Canny, J., Malik, J., Edwards, D., 1995. Artificial Intelligence: A Modern Approach. Retrieved from. https://www.cis.uab. edu/courses/cs760/Spring-660-2007/7A-660-PLUS-SYLLABUS-DRAFT.

pdf. Schulzrinne, H., Casner, S.L., Frederick, R., Jacobson, V.. RTP: A Transport Protocol for

Real-Time Applications (No. 3550). RFC 3550. RFC Editor. Retrieved from. https://rfc-editor.org/rfc/rfc3550.txt.

Schwartz, B., 2017. Scalability Is Quantifiable: The Universal Scalability Law. Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L., 2016. Edge computing: vision and challenges.

IEEE Internet Things J. 3 (5), 637–646. Siegle, J.H., Lopez, A.C., Patel, Y.A., Abramov, K., Ohayon, S., Voigts, J., 2017. Open

ephys: an open-source, plugin-based platform for multichannel electrophysiology. J. Neural Eng. 14 (4), 045003.

Sørensen, K., Neumann, C., Dahne, M., Hansen, K.A., Wahlberg, M., 2020. Gentoo penguins (Pygoscelis papua) react to underwater sounds. R. Soc. Open Sci. 7 (2), 191988.

L.B. Larsen et al.

Ecological Informatics 63 (2021) 101290

10

Stidsholt, L., Johnson, M., Beedholm, K., Jakobsen, L., Kugler, K., Brinkløv, S., Madsen, P. T., 2019. A 2.6-g sound and movement tag for studying the acoustic scene and kinematics of echolocating bats. Methods Ecol. Evol. 10 (1), 48–58.

Tchernichovski, O., Nottebohm, F., Ho, C.E., Pesaran, B., Mitra, P.P., 2000. A procedure for an automated measurement of song similarity. Anim. Behav. 59 (6), 1167–1176. https://doi.org/10.1006/anbe.1999.1416.

Tinbergen, N., 1963. On aims and methods of ethology. Z. Tierpsychol. 20 (4), 410–433. User Datagram Protocol (No. 768), 1980, August. RFC 768. RFC Editor. Retrieved from.

https://rfc-editor.org/rfc/rfc768.txt. 10.17487/RFC0768. Vohra, D., 2016. Apache Kafka. In: Pract. Hadoop Ecosyst. Apress, Berkeley, CA,

pp. 339–347. https://doi.org/10.1007/978-1-4842-2199-0_9. Retrieved from http://link.springer.com/10.1007/978-1-4842-2199-0{\_}9.

Werkhoven, Z., Bravin, A., Skutt-Kakaria, K., Reimers, P., Pallares, L.F., Ayroles, J., De Bivort, B.L., 2019a. The structure of behavioral variation within a genotype. BioRxiv 779363.

Werkhoven, Z., Rohrsen, C., Qin, C., Brembs, B., de Bivort, B., 2019b. Margo (massively automated real-time gui for object-tracking), a platform for high-throughput ethology. PLoS One 14 (11), e0224243.

Yadav, G., Paul, K., 2020. Architecture and security of scada systems: a review. arXiv preprint arXiv:2001.02925.

Ying, Y., Zhou, D.-X., 2006. Online regularized classification algorithms. IEEE Trans. Inf. Theory 52 (11), 4775–4788.

Ylonen, T., 1996. Ssh–secure login connections over the internet. In: Proceedings of the 6th Usenix Security Symposium, Vol. 37.

L.B. Larsen et al.