SWARMING PIXEL TRACKER

B.Tech Project submitted by

Parth Kanungo

Computers and Communication Engineering,

The LNM Institute of Information Technology, Jaipur

2011

CERTIFICATE It is certified that the work contained in the B.Tech. Project entitled “Swarming Pixel Tracker” by Parth Kanungo (Y07uc078) has been carried out under our supervision and that this work has not been submitted elsewhere for a degree. May, 2011 ________________________ Soumitra Debnath, Assistant Professor, Communication & Computer Engg. The LNMIIT, Jaipur

________________________ Prithwijit Guha, Visiting Faculty,

Communication & Computer Engg. The LNMIIT, Jaipur

ABSTRACT

Tracking moving objects is a critical task in computer vision, with many practical applications such as vision based interface tasks, visual surveillance or perceptual intelligence applications. This project presents a method of visual tracking using a swarm intelligence metaphor. Swarm Intelligence refers to the collective intelligent emergent behaviour of simpler agents. This is a concept inspired by the social behaviour of insects. Each agent (or swarm particle) is "unintelligent" and follows a set of primitive rules like "stay close to a neighbour". The mutual interactions and movements make the entire system "intelligent" and robust for various applications. For object tracking, we have assigned a tracker (i.e. a swarm particle) to each pixel of the object read in the first frame. As the subsequent frames of video come by, the motion of the trackers (or the swarm particles) is governed by their neighbours and their respective confidence values. The interactions of individual particles results in an emergent object tracking behaviour.

ACKNOWLEDGEMENT

I would like to express my sincere gratitude to Prof. Prithwijit Guha for guiding and supporting

me throughout the duration of this project. I would also like to thank my friend Dushyant Goyal

for valuable discussions.

CONTENTS

1. Introduction to Swarms

a. Introduction

b. Mathematical Modelling of Swarm

2. Swarm Simulation

a. The Data Structure

b. Initialize a swarm

c. Step response of a swarm

d. Assigning goals to the swarm

e. Behaviour in a two-tank system

3. Application in Visual Tracking

a. Introduction

b. Mathematical Model

c. Results

4. Conclusions and Future Work

5. References

Chapter 1

Introduction to Swarms

Flocking birds

1.1 Introduction

Birds flock, ants forage, fish school. Evolution of creatures has produced swarming in so-many

contexts. Swarming has many individual and group benefits. For example – swarming provides

ability to forage more effectively. In case of birds, it reduces the overall air drag on the birds

giving them an ability to travel greater distances. Then there is a concept of safety associated as

well. Fishes in swarms are safer compared with when they are alone. The dynamics of predator-

prey algorithm are such that the predator gets confused when it sees multiple preys at the same

time, and is unable to attack.

This concept of swarms can be applied in computers to solve problems. In fact, many problems

are ideal candidates for swarm algorithms, including traveling salesman problems, data fusion on

distributed sensor networks, network load balancing, and the coordination of large numbers of

machines, e.g., cars on the highway or automatons in a factory.

Before going into the details, let us establish a concrete definition of a swarm and an autonomous

agent. An autonomous agent is an entity that is able to sense and affect its environment. It

interacts with its environment, which probably consists of other agents, but acts relatively

independently from all the other agents. It does not follow commands from a leader, or some

global plan. Such agent is referred to as a swarm-particle (or individual) and the collection of

these particles, which produces an emergent behaviour, is referred to as a swarm.

Using the swarms in computer science, we employ what is referred to as “swarm intelligence”, a

term first used by Beni, Hackwood, and Wang in regards to the self-organizing behaviours of

cellular robotic systems [1]. This interpretation of swarm intelligence focuses on unintelligent

agents with limited processing capabilities, but possessing behaviours that collectively are

intelligent. This behaviour can be applied in optimization in telecommunicate systems, robotics,

tracking patterns in transportation systems, and military applications.

Thus, Computer Scientists have used the concept of swarm to formulate various algorithms like

Particle Swarm Optimization and Ant colony Optimization. Then, there are BOIDs, which are

simulated flocking creatures created by Craig Reynold. They have a lot of advantages in

computer science – they are robust, simple agents, distributed & parallel and effort magnification

and scalability emerges in their behaviour. [2]

In this project, instead of using these standard algorithms, we follow a more primitive approach

of implementing basic motion rules to the swarm particles, as a result of which we get emergent

swarm behaviour.

1.2 Mathematical Modelling of Swarm

Swarm is an emergent behaviour arising from simple rules that are followed by particles (or

individuals) and does not involve any central coordination. The simplest mathematical models of

swarm follow the following rules:

1. Move in the same direction as your neighbours

2. Remain close to your neighbours

3. Avoid collisions with your neighbours

This can be made possible by implementing either of the two models as shown in the figure:

The Metric Distance Model consists of three zones around an animal. In the „zone of repulsion‟,

closest to the animal, the focal animal will seek to distance itself from its neighbours to avoid

collision. In the other zone „zone of alignment‟, the particles will move in the direction of the

neighbour. In the outermost zone, the animals attract each other, so as to make the swarm

cohesive.

In the other model known as the Topological Distance Model, the animal pays attention to only 6

or 7 animals closest to itself, regardless of the distance.

To end this section, I would like to quote some lines from the novel PREY written by the

Science Fiction author Micheal Chricton:

“In the last few years, artificial life had replaced artificial intelligence as a long-term computing

goal. The idea was to write programs that had the attributes of living creatures—the ability to

adapt, cooperate, learn, adjust to change. Many of those qualities were especially important in

robotics, and they were starting to be realized with distributed processing. Distributed

processing meant that you divided your work among several processors, or among a network of

virtual agents that you created in the computer. There were several basic ways this was done.

One way was to create a large population of fairly dumb agents that worked together to

accomplish a goal—just like a colony of ants worked together to accomplish a goal……The

emergent behavior is the sum of individual agent behaviors. There isn‟t anybody there to

„decide‟ anything. There‟s no brain, no higher control in that swarm.”[3]

Chapter 2

Swarm Simulation

2.1 The Data Structure

We implemented a data structure for the swarm and its individuals. The rules were assigned to the

individual swarm particles, we called CIndividual. (In this text, we shall use CIndividual and particle

interchangeably.)

In all the experiments, we made use of a single swarm with a swarm-id and an array of individuals

that comprised the swarm. The entire simulation was done using the OpenCV Library of C++ in

Ubuntu 9.10

CSwarm

{

int sName;

int* iArray[NUM]

}

CIndividual

{

int mID; // ID of the individual

int m_X, m_Y; // location on the image (x,y)

float m_vX, m_vY; //its velocity

int r,g,b; // the r,g,b components of its color

}

This is the primitive data structure. Depending on the specific goal of the program, some more

members (like a Confidence value or the distance_from_the_goal) shall be added to the CInidvidual

data structure later.

2.2 Initialize a swarm

Swarm Intelligence is used in swarm robots to help them achieve their goals. Initially, the robots are

kept near each other. Next, they need to self-arrange themselves into random positions such that they

satisfy certain minimum and maximum distance constraints. This step is essential for the overall

coordination of the swarm-robots, so that, the robots do not go farther than the range of the sensors.

This arrangement involves an algorithm called the swarm dispersion algorithm. [4]

Here, we simply initialized a swarm with 2 constraints –

1. There should be at least one particle in the neighbourhood of another particle.

2. The particles should not overlap, or else the overall number of swarm particles would get

reduced.

Thus, we initialized a swarm of N particles; we randomly picked up an N (100 in most of the

experiments).

Correctly initialized swarm Incorrectly initialized swarm

Precaution: Make sure that the swarm has a random shape, and not a square one. For that, assign the

location of particle at a random location around the particle. In case, we assign the

locations of all the particles with respect to the 1st particle only, the swarm will be of square shape,

which looks unrealistic.

2.2 Randomly moving swarm

Once we have initialized a swarm, the next step is to start the motion of the swarm, and make it

behave like a biological system. We will initialize the various forces amongst the particles,

which will result in an emergent random behaviour of the swarm. We shall introduce these forces

in the upcoming sections.

2.3 Step Response of a swarm

To observer the step response of the swarm, we select a peripheral element and give it a velocity and

see how the entire swarm responds to this stimulus.

On successful simulation of the step response, we observe the “subway effect”, which is the gradual

propagation of information to the swarm particles.

To understand the term, imagine a crowded subway. On a crowded subway, when the people at the

forefront walk, the entire crowd does not walk simultaneously. It happens gradually. Like initially

only first 2 rows start moving, then next 2, then next 3 and so on. It takes some time for the effect of

people walking at the forefront to reach the people at the last. This slow propagation of speed is the

„subway effect‟. In the same way, under the influence of stimuli, the velocities of the swarm particles

changes gradually and hence we observe the “subway effect.”

First step in this process is to find out a peripheral particle and pass on some stimuli to it.

2.3.1 To find the peripheral particle

We shall use the concept of SCAN LINE. To find the leftmost particle of the swarm, we shall start

scanning the image. Imagine a vertical line (called the Scan Line) spanning the entire height of the

image positioned at x=0.

We move this scan line from x = 0 to x = width_of_the_image, until we get a white pixel (or a

particle). As soon as the scan line detects this first particle, we break out of the loop, and report the

location of that particle to the main code.

The red line in the figure indicates the scan line, which travels from left to right and finds out the left

peripheral particle of the swarm.

2.3.2 The Mathematical Model

Once we impart the stimuli to the peripheral element, we need to establish mathematical relationships

among the other particles, so that, this stimuli can be propogated. Following is the model followed

for this:

Let the number of particles in the swarm be N.

Region of influence around a particle = (2M+1)*(2M+1)

Let be the velocity of the neighbour of a particle, and there are r neighbours around the

particle.

Then velocity of a particle is given by

where

Where is the euclidean distance between the particle and its neighbours

2.3.3 Important Points:

1. We need to assign some potential (which causes repulsion) at the boundaries. Otherwise, the

swarm may go out of the screen leading to Segmentation Fault, OR it may go to the left and

emerge from the right of the screen.

2. For keeping track of neighbours, we can introduce a new member in our existing CIndividual

data structure, i.e., an array to neighbours, using which we can calculate the neighbourhood

adjacency matrix.

2.3.4 RESULTS

Images 1 – 6 show the various stages of the Step Response:

1 2

3 4

5 6

2.4 Assigning Goal to the swarm

2.4.1 The Birds’ Analogy

This system is similar to the swarm of birds searching for food. Each bird is following the

primitive rules of attraction and repulsion. In addition, each bird is exploiting the eyes of every

other bird, by estimating their distance from the food, and following the one which is closest to

the food.

The simulation of this efficient approach is the subject of this subsection. Here, we show how

we implemented attraction and repulsion among the swarm elements, and their urge to reach the

goal.

2.4.2 Mathematical model

Attraction is necessary so to keep the swarm cohesive, otherwise there is a possibility of the

swarm getting split into two. Repulsion is needed in order to avoid collision of the swarm

particles. And, the urge is required to direct the swarm towards the goal (food).

[Attraction factor]

[Repulsion factor]

[Urge to reach the goal]

Here d1 is the distance between the swarm particles, and d2 is the distance of the swarm particle

from the goal.

The typical values that were suitable for animation were found after a lot of trial-and-error. Thus,

the values we used for these constants are

And

Where and denotes respectively the new and the old location of the swarm-particles.

We introduced attraction in the 5x5 neighbourhoods and repulsion in the 3x3 neighbourhoods.

Precaution: One problem that may arise during this process is that different characteristics can

interfere with each other. Too much attraction can drive to lose track of its goal. Too much

repulsion can break the swarm apart. Hence, the values mentioned above came after a number of

experiments.

2.4.3 Results

Our goal coordinates were = (300,200). Interestingly, the swarm takes a V kind of a shape in this

simulation.

2.5 Behaviour in a 2 tank system.

In this subsection, we will discuss the modifications needed to help the swarm travel through long

narrow corridors. The scenario is as follows:

The swarm has to move from region A to region B without colliding with the walls of the region C.

For a single agent, we can employ the path finding algorithms like the A* algorithm [10], which is

widely used in many computer games. However, for a swarm which learns and evolves, we need a

different approach.

We define new parameters to carry out this process. We define 2 global variables –

1. The centroid of the swarm

∑

∑

2. The radius of the swarm – It is the radius of the circle circumscribing the swarm.

In the simulation, the first step is detection of the wall. As the wall is coloured grey, its detection is

quite easy. Once the wall is detected, the wall potential will not allow the swarm to pass through it.

As soon as the swarm reaches the opening of the narrow corridor, it will reduce its radius and travel

through the tunnel. Once out of the tunnel, it can go back to its original radius, and reach the target.

We can control the radius by varying the values. And, the simulation stops when the centroid

of the swarm reaches the target area.

For simulation of walls here, we can also make use of the virtual swarm agents [6].

Chapter 3

Application in Visual Tracking

3.1 Introduction

The concept of swarm intelligence can be used to track objects [5]. We can employ algorithms like

predator-prey tracking the objects, wherein the object is the prey and the swarm particles which track it

are the predators.

The basic idea is that a swarm of particles fly around the image and learns and evolves as the frames go

by. The tracking window actually encloses the whole of the swarm, thereby detecting the moving object.

We applied the algorithm on the PETS2000 dataset and detected the various moving objects in the

dataset.

3.2 Mathematical Model

Here we are using the same concept with what we define as “CONFIDENCE VALUES”.

1. The object to be detected is initialized in the first frame. The entire object is marked white as

shown in the figure. These white pixels are actually the swarm elements which will act as tracker

pixels.

2. We define “confidence value”, which represents the confidence level. They were assigned on a

scale of 1 to 10. (1 representing min, 10 representing max)

Assume that in the frame, we have a red pixel at a position . When the new frame is

loaded, we search the neighbourhood of for the same red pixel in the frame. Let denote the

number of such red pixels detected in the neighbourhood.

While searching, there are 3 outcomes possible:

1. M = 1: which means there is exactly one red pixel in the neighbourhood of the old location of the

red pixel. Hence, we have found the new position of the swarm particle representing the red pixel.

Moreover, I can calculate the velocity of the swarm particle from this data. And, I set the

confidence value of this particle as 10.

2. M = 0: Confidence value is set to zero. The particle is not sure where to go. Hence, in this case it

shifts closer to the centroid of the swarm.

3. M > 1: for this case, we use the reduced „confidence values‟. The confidence values are reduced

by using the variance of confidence values of the neighbours.

After assigning the individual confidence values, we calculate the velocities of individual swarm particle

p as

,

where ‟s are the “Confidence values” of the particles.

The process is repeated for each swarm particle in each new frame, and we observer the emergent object

tracking behaviour.

(The swarm particles covering the car, which help in object detection.)

3.3 RESULTS

Chapter 4

Conclusions and Future Work

Swarming Pixel Tracker is an intelligent system of tracking objects. However, a lot of

improvements and analysis are possible.

1. Stability of swarms is a critical issue. We need to assign proper values of attraction and

repulsion factors. A slight incorrectly estimated value may result in the swarm breaking

away into fragments, or collapsing into its centre. This issue of stability has although

been studies [8, 9], yet it still remains an open problem.

2. We have manually detected the object in the first frame. This detection can be

automatized.

3. We found that in various applications, the actual number of particles in the swarm does

not matter, as long as the number is large enough to get our goal accomplished. Hence, in

our applications we may give rise to or kill the swarm particles.

4. Real-time simultaneity is not easily achievable. This is a limitation of the simulation

which can be overcome, if we build better algorithms for swarm simulation.

5. Predicting the exact behaviour of the swarm is also a problem that can be studied.

6. As long as the swarm particles accomplish the assigned goal, the actual number of

particles in a swarm does not matter. In many cases, they may die or new ones may take

birth.

7. Research is possible in the swarm simulation also. We can study how 2 or 3 swarms

interact with each other. That observation will provide us a deeper insight into the swarm

behaviour.

8. Swarming algorithms can be used for various life-like simulations. It is here to stay for

long. This research can be further expanded towards building a road traffic model.

Chapter 5

References

1. Beni, G., Wang, J. Swarm Intelligence in Cellular Robotic Systems, Proceed. NATO

Advanced Workshop on Robots and Biological Systems, Tuscany, Italy, June 26–30

(1989).

2. Reynolds CW (1987) "Flocks, herds and schools: A distributed behavioural model"

Computer Graphics, 21(4), 25–34

3. Prey by Micheal Chricton

4. MA KOVACINA - SWARM ALGORITHMS: SIMULATION AND GENERATION

5. Swarm Intelligence Based Dynamic Object Tracking – by Yuhua Zheng and Yan Meng.

2008 IEEE Congress on Evolutionary Computation (CEC 2008)

6. Exerting Human Control Over Decentralized Robot Swarms by Zsolt Kira and Mitchell

A. Potter

7. Swarm Intelligence: Literature Overview by Yang Liu and Kevin M. Passino, The Ohio

State University

8. K. Jin, P. Liang, and G. Beni, Stability of synchronized distributed control of discrete

swarm structures," in IEEE International Conference on Robotics and Automation, pp.

1033{1038, 1994}

9. G. Beni and P. Liang, Pattern reconfiguration in swarms {convergence of a distributed

asynchronous and bounded iterative algorithm," IEEE Trans. on Robotics and

Automation, vol. 12, June 1996

10. http://www.gamasutra.com/view/feature/3096/toward_more_realistic_pathfinding.php