1

VISION BASED SHAPE IDENTIFICATION AND POSITIONING (Robotic Arm)

ADVANCED ROBOTICS

ASSIGNMENT

2

Table of Contents

TITLE…………………………………………………………………………………………………1

OBJECTIVES……………………………………………………………………………….………3

INTRODUCTION……………………………………………………………………………..….4

REQUIREMENTS………………………………………………………………………………17

METHOD…………………………………………………………………………………………17

PROGRAM…………………………………………………………………………………….…41

RESULTS…………………………………………………………………………………….……42

IMPROVEMENTS…………………………………………………………………………..…44

CONCLUSION……………………………………………………………………………..……44

REFERENCE………………………………………………………………………………………45

3

TITLE:

Vision based shape identification and position

OBJECTIVES:

Using Roborealm software the shape classification must be identified. The Lynx motion arm robot must

perform pick and place operation using this information. (Was not able to perform pick and place as

software crashed again and again, only could move the arm to a particular point)

Identifying the shapes:

i) The first objective is to identify shapes and should be classified using different values. This

should be done in a single program.

ii) The shapes identification algorithm should not be sensitive to orientation or position.

iii) The shapes identification algorithm should also not be sensitive to size variation.

Shape position:

i) Once the shape has been identified the coordinates must be transfer to the robot, to perform

pick and place operation (Was not able to pick and place as more than 1 lynx motion command

crashed in the roborealm software)

Student will need to come up with one application method for image classification. Detailed explanation

must be given on this method with proper calculation if necessary.

4

INTRODUCTION (Detailed):

1. Roborealm:

It is a powerful robotic vision software application for use in computer vision, image processing,

and robot vision tasks. Using an easy to use point and click interface, complex image analysis

and robot control becomes easy.

Theory

The main purpose of roborealm is to translate what a camera sees into meaningful numbers in

order to cause a reaction to what the image contains. This requires the processing of an image

into a few or a single number that is meaningful in the current context of whatever project you

have in mind. As each project has vastly different requirements with different attributes that

need to be detected, roborealm is written in a flexible approach where the basic tools are

provided to you in the form of modules in order to be combined into a relevant combination.

For example, if your task is to track a red ball then the color red can be picked up by a color filter

module as the first step. But if you wanted to track any type of ball then you would not use the

red color as an attribute but instead a round shape detector. The reason you would ever want to

use a color tracker instead of just using a shape tracker is that the color tracker may be more

reliable and repeatable in some environments. Some features of roborealm are as,

Inexpensive Vision Application

Links Vision to Motion

Interactive GUI Interface

Socket based Server API

Multiple Interfaces (Disk, Web, FTP, Email, etc)

Plug-in Framework for Custom Modules

5

Compatibility

Roborealm is compatible with the PCTx, Servo Controller, and Analog Readers for servo and

motor control and sensor acquisition as well as many other devices from various manufacturers.

2. Vision System:

Computer based device for interpreting visual signals from a video camera. Computer vision is

important in robotics where sensory abilities would considerably increase the flexibility and

usefulness of a robot. Vision systems very often use pattern recognition and are used in a

variety of applications, from automated stock control robots to quality control in automated

manufacturing systems.

It is the science and technology of machines that see. As a scientific discipline, computer vision

is concerned with the theory behind artificial systems that extract information from images. The

image data can take many forms, such as video sequences, views from multiple cameras, or

multi-dimensional data from a medical scanner.

As a technological discipline, computer vision seeks to apply its theories and models to the

construction of computer vision systems. Examples of applications of computer vision include

systems for:

Controlling processes (e.g. an industrial robot or an autonomous vehicle)

Detecting events (e.g. for visual surveillance or people counting)

Organizing information (e.g. for indexing databases of images and image sequences)

Modeling objects or environments (e.g. industrial inspection, medical image analysis or

topographical modeling)

Interaction (e.g. as the input to a device for computer-human interaction)

6

Computer vision is closely related to the study of biological vision. The field of biological vision studies

and models the physiological processes behind visual perception in humans and other animals.

Computer vision, on the other hand, studies and describes the processes implemented in software and

hardware behind artificial vision systems. Interdisciplinary exchange between biological and computer

vision has proven fruitful for both fields. Computer vision is, in some ways, the inverse of computer

graphics. While computer graphics produces image data from 3D models, computer vision often

produces 3D models from image data. There is also a trend towards a combination of the two

disciplines, e.g., as explored in augmented reality. Sub-domains of computer vision include scene

reconstruction, event detection, video tracking, object recognition, learning, indexing, motion

estimation, and image restoration.

The fields most closely related to computer vision are image processing, image analysis and machine

vision. There is a significant overlap in the range of techniques and applications that these cover. This

implies that the basic techniques that are used and developed in these fields are more or less identical,

something which can be interpreted as there is only one field with different names. On the other hand,

it appears to be necessary for research groups, scientific journals, conferences and companies to

present or market themselves as belonging specifically to one of these fields and, hence, various

characterizations which distinguish each of the fields from the others have been presented.

The following characterizations appear relevant but should not be taken as universally accepted:

Image processing and image analysis tend to focus on 2D images, how to transform one image

to another, e.g., by pixel-wise operations such as contrast enhancement, local operations such

as edge extraction or noise removal, or geometrical transformations such as rotating the image.

This characterization implies that image processing/analysis neither require assumptions nor

produce interpretations about the image content.

Computer vision tends to focus on the 3D scene projected onto one or several images, e.g. how

to reconstruct structure or other information about the 3D scene from one or several images.

Computer vision often relies on more or less complex assumptions about the scene depicted in

an image.

7

Machine vision tends to focus on applications, mainly in manufacturing, e.g. vision based

autonomous robots and systems for vision based inspection or measurement. This implies that

image sensor technologies and control theory often are integrated with the processing of image

data to control a robot and that real-time processing is emphasized by means of efficient

implementations in hardware and software. It also implies that the external conditions such as

lighting can be and are often more controlled in machine vision than they are in general

computer vision, which can enable the use of different algorithms.

There is also a field called imaging which primarily focus on the process of producing images,

but sometimes also deals with processing and analysis of images. For example, medical imaging

contains lots of work on the analysis of image data in medical applications.

Finally, pattern recognition is a field which uses various methods to extract information from

signals in general, mainly based on statistical approaches. A significant part of this field is

devoted to applying these methods to image data.

Typical tasks of computer vision

Each of the application areas employ a range of computer vision tasks; more or less well-defined

measurement problems or processing problems, which can be solved using a variety of methods. Some

examples of typical computer vision tasks are presented below.

Recognition

The classical problem in computer vision, image processing, and machine vision is that of determining

whether or not the image data contains some specific object, feature, or activity. This task can normally

be solved robustly and without effort by a human, but is still not satisfactorily solved in computer vision

for the general case: arbitrary objects in arbitrary situations. The existing methods for dealing with this

problem can at best solve it only for specific objects, such as simple geometric objects (e.g.

polyhedrons), human faces, printed or hand-written characters, or vehicles, and in specific situations,

typically described in terms of well-defined illumination, background, and pose of the object relative to

the camera.

8

Different varieties of the recognition problem are:

Object recognition: One or several pre-specified or learned objects or object classes can be

recognized, usually together with their 2D positions in the image or 3D poses in the scene.

Identification: An individual instance of an object is recognized. Examples: identification of a

specific person's face or fingerprint, or identification of a specific vehicle.

Detection: The image data is scanned for a specific condition. Examples: detection of possible

abnormal cells or tissues in medical images or detection of a vehicle in an automatic road toll

system. Detection based on relatively simple and fast computations is sometimes used for

finding smaller regions of interesting image data which can be further analyzed by more

computationally demanding techniques to produce a correct interpretation.

Motion analysis

Several tasks relate to motion estimation where an image sequence is processed to produce an estimate

of the velocity either at each point in the image or in the 3D scene, or even of the camera that produces

the images. Examples of such tasks are:

Ego motion: Determining the 3D rigid motion (rotation and translation) of the camera from an

image sequence produced by the camera.

Tracking: Following the movements of a (usually) smaller set of interest points or objects (e.g.,

vehicles or humans) in the image sequence.

9

Optical flow: To determine, for each point in the image, how that point is moving relative to the

image plane, i.e. its apparent motion. This motion is a result both of how the corresponding 3D

point is moving in the scene and how the camera is moving relative to the scene.

Scene reconstruction

Given one or (typically) more images of a scene, or a video, scene reconstruction aims at computing a 3D

model of the scene. In the simplest case the model can be a set of 3D points. More sophisticated

methods produce a complete 3D surface model.

Image restoration

The aim of image restoration is the removal of noise (sensor noise, motion blur, etc.) from images. The

simplest possible approach for noise removal is various types of filters such as low-pass filters or median

filters. More sophisticated methods assume a model of how the local image structures look like, a model

which distinguishes them from the noise. By first analyzing the image data in terms of the local image

structures, such as lines or edges, and then controlling the filtering based on local information from the

analysis step, a better level of noise removal is usually obtained compared to the simpler approaches.

Computer vision systems

The organization of a computer vision system is highly application dependent. Some systems are stand-

alone applications which solve a specific measurement or detection problem, while others constitute a

sub-system of a larger design which, for example, also contains sub-systems for control of mechanical

actuators, planning, information databases, man-machine interfaces, etc. The specific implementation

of a computer vision system also depends on if its functionality is pre-specified or if some part of it can

be learned or modified during operation. There are, however, typical functions which are found in many

computer vision systems.

10

Image acquisition: A digital image is produced by one or several image sensors, which, besides

various types of light-sensitive cameras, include range sensors, tomography devices, radar,

ultra-sonic cameras, etc. Depending on the type of sensor, the resulting image data is an

ordinary 2D image, a 3D volume, or an image sequence. The pixel values typically correspond to

light intensity in one or several spectral bands (gray images or color images), but can also be

related to various physical measures, such as depth, absorption or reflectance of sonic or

electromagnetic waves, or nuclear magnetic resonance.

Pre-processing: Before a computer vision method can be applied to image data in order to

extract some specific piece of information, it is usually necessary to process the data in order to

assure that it satisfies certain assumptions implied by the method. Examples are

Re-sampling in order to assure that the image coordinate system is correct.

Noise reduction in order to assure that sensor noise does not introduce false

information.

Contrast enhancement to assure that relevant information can be detected.

Scale-space representation to enhance image structures at locally appropriate scales.

Feature extraction: Image features at various levels of complexity are extracted from the image

data. Typical examples of such features are

Lines, edges and ridges.

Localized interest points such as corners, blobs or points.

Detection/segmentation: At some point in the processing a decision is made about which image

points or regions of the image are relevant for further processing. Examples are

11

Selection of a specific set of interest points

Segmentation of one or multiple image regions which contain a specific object of

interest.

High-level processing: At this step the input is typically a small set of data, for example a set of

points or an image region which is assumed to contain a specific object. The remaining

processing deals with, for example:

Verification that the data satisfy model-based and application specific assumptions.

Estimation of application specific parameters, such as object poses or objects size.

Classifying a detected object into different categories.

3. Image Processing:

Image processing is a physical process used to convert an image signal into a physical image. The

image signal can be either digital or analog. The actual output itself can be an actual physical

image or the characteristics of an image.

The most common type of image processing is photography. In this process, an image is

captured using a camera to create a digital or analog image. In order to produce a physical

picture, the image is processed using the appropriate technology based on the input source

type.

In digital photography, the image is stored as a computer file. This file is translated using

photographic software to generate an actual image. The colors, shading, and nuances are all

12

captured at the time the photograph is taken, the software translates this information into an

image.

When creating images using analog photography, the image is burned into a film using a

chemical reaction triggered by controlled exposure to light. The image is processed in a

darkroom, using special chemicals to create the actual image. This process is decreasing in

popularity due to the advent of digital photography, which requires less effort and special

training to product images.

In addition to photography, there are a wide range of other image processing operations. The

field of digital imaging has created a whole range of new applications and tools that were

previously impossible. Face recognition software, medical image processing and remote sensing

are all possible due to the development of digital image processing. Specialized computer

programs are used to enhance and correct images. These programs apply algorithms to the

actual data and are able to reduce signal distortion, clarify fuzzy images and add light to an

underexposed image.

Image processing techniques were first developed in 1960 through the collaboration of a wide

range of scientists and academics. The main focus of their work was to develop medical imaging,

character recognition and create high quality images at the microscopic level. During this period,

equipment and processing costs were prohibitively high.

The financial constraints had a serious impact on the depth and breadth of technology

development that could be done. By the 1970s, computing equipment costs had dropped

substantially making digital image processing more realistic. Film and software companies

invested significant funds into the development and enhancement of image processing, creating

a new industry.

There are three major benefits to digital image processing. The consistent high quality of the

image, the low cost of processing and the ability to manipulate all aspects of the process are all

great benefits. As long as computer processing speed continues to increase while the cost of

storage memory continues to drop, the field of image processing will grow.

13

Digital image processing is the use of computer algorithms to perform image processing on

digital images. As a subfield of digital signal processing, digital image processing has many

advantages over analog image processing; it allows a much wider range of algorithms to be

applied to the input data, and can avoid problems such as the build-up of noise and signal

distortion during processing.

Digital image processing allows the use of much more complex algorithms for image processing,

and hence can offer both more sophisticated performance at simple tasks, and the

implementation of methods which would be impossible by analog means. In particular, digital

image processing is the only practical technology for:

Classification

Feature extraction

Pattern recognition

Projection

Multi-scale signal analysis

Some techniques which are used in digital image processing include:

Pixelization

Linear filtering

Principal components analysis

Independent component analysis

Hidden Markov models

Partial differential equations

Self-organizing maps

Neural networks

Wavelets

14

Finally summing it up, Image processing converts a target image captured by a Charged Couple Device

(CCD) camera into a digital signal and then performs various arithmetic operations on the signal to

extract the characteristics of the target, such as area, length, quantity and position. Finally, a good result

is output based on preset tolerance limits.

15

4. Some applications of vision systems and image processing:

Computer vision (I)

Optical sorting (I)

Augmented Reality (I)

Face detection (I)

Feature detection (I)

Lane departure warning system (I)

Non-photorealistic rendering (I)

Medical image processing (I)

Microscope image processing (I)

Morphological image processing (I)

Remote sensing (I)

Vision System Measures Scallops (V)

Inspecting turbine blades in aircraft engines (V)

Vision automates parking surveillance (V)

Vision helps delta robot sort biscuits (V)

Laser Marking and Image-based Industrial ID Reader Save Hundreds of Thousands of Dollars (V)

Automotive Supplier Achieves Perfect Quality with Low-Cost Machine Vision (V)

Vision System Prevents Injection Molding Tool Damage and Improves Part Quality (V)

Inspection of band saw blades made easy (V)

5. Robot Vision (Optional – Summing up for Robots):

The field of robot vision guidance is developing rapidly. The benefits of sophisticated vision

technology include savings, improved quality, reliability, safety and productivity. Robot vision is

used for part identification and navigation. Vision applications generally deal with finding a part

and orienting it for robotic handling or inspection before an application is performed.

Sometimes vision guided robots can replace multiple mechanical tools with a single robot

station.

16

Creating Sight

A combination of vision algorithms, calibration, temperature software, and cameras provide the

vision ability. Calibration of robot vision system is very application dependent. They can range

from a simple guidance application to a more complex application that uses data from multiple

sensors.

Algorithms are consistently improving, allowing for sophisticated detection. Many robots are

now available with collision detection, allowing them to work alongside other robots without

the fear of a major collision. They simply stop moving momentarily if they detect another object

in their motion path.

Seeing Savings

Robotic vision makes processes simpler, more straightforward, thus cutting costs:

Fixtures: Robot vision eliminates any need for hard tooling or fixtures. Now, products can be

identified and applications performed without any need for securing.

Labor: There are labor and machinery savings that come with robotic vision. There is no need for

sorts, feeders or upstream actuators anymore. Nor is there any need for labor to load or orient

parts.

Finding the Right Vision

When deciding on the right robot vision guidance, work with an integrator you can trust and

consider the following:

Communication: Robot vision must work and connect with the robot system and application. A

disconnect could harm the robot or the product and cause loss of production and quality.

17

Environment: The workplace must be controlled so that robot vision remains sharp. Every

contributing element in the environment, including lighting, product color changes, airborne

chemicals, must be considered and tested.

REQUIREMENTS:

1. Roborealm Software (For Image processing)

2. 5 Images (Square, Circle, Triangle, Rectangle and all together)

3. Camera (For Vision)

4. Lynx motion SSC – 32 (Robotic arm)

METHOD:

Firstly we need to install roborealm into the PC, and then connect the USB webcam to the PC. After that

we start our roborealm software. Go to the options and select video and input your camera’s name in

the camera text box. After finishing this, go to main screen and press camera tab, doing so the camera

will start working and you can see your image in the roborealm work screen. Now we take the all in one

image (having square, circle, triangle and rectangle together) and put it under the camera. So now we

can see the image on the roborealm work screen. So we see that our vision system has worked. That

means our camera has worked and is connected to the software. Now comes the part of image

processing. So, as we have the image already on the work screen, we go further with image processing

coding (Filtering). As the first step of image processing we,

1. Make our image inverted using the Negative module under adjust

A positive image is a normal image. A negative image is a tonal inversion of a positive image, in

which light areas appear dark and vice versa. A negative color image is additionally color

reversed, with red areas appearing cyan, greens appearing magenta and blues appearing yellow.

18

Film negatives usually also have much less contrast than the final images. This is compensated

by the higher contrast reproduction by photographic paper or by increasing the contrast when

scanning and post processing the scanned images.

The Negative (Solarize) module inverts all pixel values. For example, if a pixel is white it is

changed to black, if it is black it is changed to white.

R=255-R

G=255-G

B=255-B

We use this module to sharpen up the image, remove noise and smoothen it up. Using this filter

we can clearly specify the object in the image separately. It helps us to know our objects and

background.

(The normal image which we put below the camera)

19

(This is the image we get after using negative module on the original image)

Seeing the image we can clearly identify the objects and the background. It makes the image

sharper and easy to identify.

20

2. After this we use the Auto Threshold module under threshold

Thresholding is the simplest method of image segmentation. From a grayscale image,

thresholding can be used to create binary images

The Auto Threshold modules will automatically threshold the current image into a binary black

and white image. The threshold level is automatically determined based on the method

selected. The appropriate method to use will depend on your application. Select Cluster (Otsu) if

you are looking for a standard technique that is most often referenced by the current machine

vision literature.

This module is useful when working with blob analysis or shape recognition whose background

image can change and a manual threshold is not reliable enough.

The following briefly outlines the algorithms used by the thresholding methods to allow you to

choose the most appropriate for your application. Note that they all operate on the image's

histogram.

Two Peaks - Detects the two highest peaks in the histogram separated by the distance specified.

The distance will ensure that peaks close to each other are not selected. The threshold is then

found by finding the deepest valley between these two peaks.

Mean Level - the average pixel value is determined using the image histogram. All pixel

intensities below that value are set to black with all pixel intensities above the mean set to

white.

Black Percent - Also known as P-Tile. The threshold level is set based on the specified percent of

suggested dark pixels (or background) there are in the image. The histogram is used to indicate

how much of the image at a certain threshold would be set to black. Once this amount exceeds

the specified percent the current histogram index (0-255) is used as the threshold.

Edge Percent - Similar to the black percent the edge percent threshold is determined by the

specified percent of edge pixels that exist below the threshold. Instead of just counting every

pixel the edge percent basis its measurement on how much a pixel is part of an edge by

performing a laplacian filter prior to threshold determination.

21

Entropy (Kapur) - Utilizes Kapur's entropy formula to find the threshold that minimizes the

entropy between the two halves of the histogram created by a threshold.

Cluster (Otsu) - One of the most popular threshold techniques that creates two clusters (white

and black) around a threshold T and successively tests the within-class variance of the clusters

for a minimum. This algorithm can also be thought of as maximizing the between-class

variance.

Symmetry - Assumes that the largest peak in the histogram is somewhat symmetrical and uses

that symmetry to create a threshold just before the beginning of the largest peak. This

technique is particularly useful to segment objects from large background planes.

Triangle - Works well with histograms that don't have well defined peaks. This technique finds

the maximum distance between a suggested threshold value and a line that connects the first

non-zero pixel intensity with the highest peak. Inherent in this technique is the distance of a

point to a line equation.

For our image we use the Otsu algorithm of the auto threshold module. In computer vision and

image processing, Otsu's method is used to automatically perform histogram shape-based

image thresholding or the reduction of a gray level image to a binary image. The algorithm

assumes that the image to be threshold contains two classes of pixels (e.g. foreground and

background) then calculates the optimum threshold separating those two classes so that their

combined spread (intra-class variance) is minimal. The extension of the original method to

multi-level thresholding is referred to as the Multi Otsu method. In Otsu's method we

exhaustively search for the threshold that minimizes the intra-class variance, defined as a

weighted sum of variances of the two classes:

Weights ωi are the probabilities of the two classes separated by a threshold t and variances

of these classes.

Otsu shows that minimizing the intra-class variance is the same as maximizing inter-class

variance:

22

Which is expressed in terms of class probabilities ωi and class means μi which in turn can be

updated iteratively. This idea yields an effective algorithm.

Algorithm

Compute histogram and probabilities of each intensity level

Set up initial ωi(0) and μi(0)

Step through all possible thresholds maximum intensity

Update ωi and μi

Compute

Desired threshold corresponds to the maximum

23

(This is the image got after applying auto threshold module to the previous image)

3. After this we use the Blob filter under blobs

The blob filter module (also known as Particle Filter or Analysis) provides a way to identify a

particular blob based on features that describe aspects of the blob itself and in relation to other

blobs. The purpose of the blob filter is to provide enough feature descriptions of the desired

blob to ensure that the identification of a particular blob is reliable and dependable despite a

noisy background.

24

The blob filter must be used after a blob segmentation module like the RGB Filter, Segment

Colors, Flood Fill, Threshold, etc. modules that will group pixels in some meaningful way into

blobs of a single color with black pixels being the background. The module you will use to

perform this segmentation will depend on your particular project task. Once the image has been

grouped into blobs this Blob Filter module is then used to remove or filter those blobs remaining

in the image that are not wanted. For example, if you have an image that was detected for the

red color using the RGB Filter module and the image included a red or orange cone the blob

filter can be used to remove all blobs that are too small and not triangular shaped in the image.

Thus any red dirt or tree bark while present after the red color detection would be removed by

using the blob filter as they would fail a triangular shape test (assuming this is one of the

attributes filtered on).

Once you have your images segmented into various blobs you then add in each blob attribute

seen below and specify a weight threshold or count threshold to remove those unwanted blobs.

Keep in mind that you can add multiple attributes one after the other that will remove blobs

along the way in order to finish with the final desired blob. Look for attributes that create a wide

distinction between your desired blob and other unwanted blobs (see the Show Weights

checkbox to see all weights given the current attribute). Using the checkbox Create Blob Array

will create the final BLOBS array variable that will contain the COG (center of gravity) of the

blobs left after the filtering process. This variable can then be used to react to the presence of a

particular object.

This will allow us to filter out the unwanted shapes. Firstly we start with a triangle. We just need

our work screen to show the image of triangle and filter out the rest of the shapes. So for doing

that we use triangle deviation (under shapes) with weight threshold of >=0.65. So that when we

put 4 shapes under the camera with blob filter still open we should just see triangle. And if we

put any other shape it should not be visible. In a way we are constraining. If any shape has

triangle deviation with weight threshold of >=0.65 then only we’ll see triangle otherwise won’t

see anything.

Triangle Deviation: Estimates a perfect triangle from blob outline and then determines

how well the blob’s outlines deviates from the ideal extracted triangle.

25

(We are just able to see the triangle as other shapes are filtered out)

26

Secondly we filter for a circle. We just need our work screen to show the image of circle and

filter out the rest of the shapes. So for doing that we use circular deviation (under shapes) with

weight threshold of >=0.89 and circular with weight threshold of >=0.78. So that when we put 4

shapes under the camera with blob filter still open we should just see circle. And if we put any

other shape it should not be visible. In a way we are constraining. If any shape has circular

deviation with weight threshold of >=0.89 and circular with weight threshold of >=0.78 then

only we’ll see circle otherwise won’t see anything.

Circular: Circular blobs get higher weights.

Circular deviation or variance: The circular variance provides a measure of the spread of a set of

dihedral angles. It is applied here to each residue's distribution of Fi, ∅i, c1, c2 and angles

across all the members of the NMR ensemble. So, for example, it can provide a measure of how

tightly or loosely a given residue's ∅ torsion angles cluster together across the entire ensemble

of models.

It is defined as

Where, , n being the number of members in the ensemble and R is given by

The expression

27

The value of the circular variance varies from 0 to 1, with the lower the value the tighter the

Clustering of the values about a single mean value.

For two dimensional distributions, such as the distributions of the - values on the residue-by-

Residue Ramachandran plots, the expression for above is modified to

28

(We are just able to see the circle as other shapes are filtered out)

Thirdly we filter for a rectangle. We just need our work screen to show the image of rectangle

and filter out the rest of the shapes. So for doing that we use circular deviation (under shapes)

with weight threshold of >=0.88 (inverted), quadrilateral sides with weight threshold of >=0.75

and quadrilateral area with weight threshold of >=0.8. So that when we put 4 shapes under the

camera with blob filter still open we should just see rectangle. And if we put any other shape it

should not be visible. In a way we are constraining. If any shape use circular deviation with

weight threshold of >=0.88 (inverted), quadrilateral sides with weight threshold of >=0.75 and

quadrilateral area with weight threshold of >=0.8 then only we’ll see rectangle otherwise won’t

see anything.

Quadrilateral Sides: Estimates a perfect rectangle from blob outline and then determines

squareness of a blob by comparing how well two sides of the ideal extracted rectangle compare

in length. The more rectangular the blob the more the opposing sides will match in length.

29

Quadrilateral Area : Estimates a perfect rectangle from blob outline and then compares how

well does the blob's area matches the area determined by the ideal four sides.

30

(We are just able to see the rectangle as other shapes are filtered out)

Fourthly we filter for a square. We just need our work screen to show the image of square and

filter out the rest of the shapes. So for doing that we use circular deviation (under shapes) with

weight threshold of >=0.88, quadrilateral sides with weight threshold of >=0.75 and

quadrilateral area with weight threshold of >=0.8. So that when we put 4 shapes under the

camera with blob filter still open we should just see square. And if we put any other shape it

should not be visible. In a way we are constraining. If any shape use circular deviation with

weight threshold of >=0.88, quadrilateral sides with weight threshold of >=0.75 and

quadrilateral area with weight threshold of >=0.8 then only we’ll see square otherwise won’t

see anything.

31

(We are just able to see the square as other shapes are filtered out)

32

4. After this we use Watch variables under statements

The Watch Variables module allows you to peek into the current variable state being maintained

by Roborealm. This module will list out all current variables that are accessible by other modules

and plugins.

5. After this we use the If statement under statements

The If Statement module allows you to create a condition on which the enclosed modules will

get executed or not. This is similar to the conditional statements seen in the VBScript and other

scripting languages. Using the interface you can compare Roborealm variables against other

variables or values to determine if the following modules should be executed or not.

33

For executing our program this module is very important. It constrains all the blob filters for a

particular shape together. Like for square the quadrilateral area (A2), quadrilateral sides and

circular deviation under shapes are constrained together (values shown in blob filter). So if any

shape fulfilling the above requirement would come under square.

If command is always followed by then and end – If command (Finish). We use then command

to use the speak function under audio plus, lynx motion SSC – 32 under servos under control.

The speak command would help us to know when particular shape is detected and lynx motion

SSC-32 command will help to move the arm to particular point when previous happens.

(For triangle)

(For circle)

34

(For rectangle)

(For square)

6. After this we use the Speak module under audio

The Speak module uses the Microsoft Speech 5.0 download to speak written text. This module is

useful for indicating certain states within the execution of the program when no screen is

available on the robot. There are two ways to use the speech module. You can either specify a

variable that will contain text that will be spoken or you can type in the text directly. Using a

variable is useful if you wish to change the spoken text based on a VBScript module or other

plugins.

35

So when a particular shape is detected then using If command the voice speaks the name of the

shape detected.

36

7. After this we use Lynx motion SSC – 32 command under servos which is under control

The Lynx motion SSC-32 module allows you to interface Roborealm to servos using a controller

made by Lynx motion called the SSC-32 Servo Controller. The servo controller supports up to 32

channels of 1uS resolution. In addition the board provides synchronized movement so that all

servos will update at the same time. The board also supports 4 digital or analog inputs for

adding additional sensors to your robotic projects.

This command is used with If command. Once a shape is detected the speak module (optional)

automatically starts. After which the Lynx motion SSC 32 command comes into play, the robotic

arms are moved to a particular co-ordinates according to the shape detected.

(For triangle)

37

(For circle)

(For rectangle)

38

(For square)

So using this command the robotic arm will move to specified position when a particular shape

is detected by the camera. (Default position adjustable – starting position)

39

PROGRAM:

Have shown just one lynx motion SSC command because if I try to put more than 1 lynx motion SSC

command in the program the roborealm crashes and computer hangs. For the presentation purpose we

can use the program without the speak command.

40

RESULTS:

After making the program we tested it with the various shapes at different orientations. We got very

good results. Our program was able to detect various shapes at different orientations and spoke the

name of the shape when detected plus moved the robotic arm to the position we have input.

Circle Triangle Rectangle Square

Orientation 1 Good Good Bad Good

Orientation 2 Good Good Bad Good

Orientation 3 Good Good Good Bad

Orientation 4 Good Good Good Good

Circle Triangle Rectangle Square

Orientation 1 Good Good Bad Good

Orientation 2 Good Good Good Bad

Orientation 3 Good Good Good Good

Orientation 4 Good Good Good Good

Circle Triangle Rectangle Square

Orientation 1 Good Good Good Bad

Orientation 2 Good Good Good Good

Orientation 3 Good Good Good Good

Orientation 4 Good Good Good Good

41

Circle Triangle Rectangle Square

Orientation 1 Good Good Good Good

Orientation 2 Good Good Good Good

Orientation 3 Good Good Good Good

Orientation 4 Good Good Good Good

Circle Triangle Rectangle Square

Orientation 1 Good Good Good Good

Orientation 2 Good Good Good Good

Orientation 3 Good Good Good Good

Orientation 4 Good Good Good Good

Circle Triangle Rectangle Square

Orientation 1 100% 100% 60% 80%

Orientation 2 100% 100% 80% 80%

Orientation 3 100% 100% 100% 80%

Orientation 4 100% 100% 100% 100%

We see that testing for circle and triangle was perfect. Which means the camera could detect these 2

shapes easily and go forward with the commands decides. But this isn’t the same case with rectangle

and square. Reason being it was very difficult to separate square from rectangle. As both of them are

quadrilateral. But at end after changing the values of the blob filter we got perfect output. Which means

camera was able to detect square and rectangle separately and go forward with the commands decided.

42

IMPROVEMENTS:

We can use Bitmap under blob filter along the present program to have a more accurate output. Or we

can use shape matching (with COG under analysis) under matching along with the present program and

constrain its variables using the If command to the present If command. Doing so we would have a

higher rate of accuracy. Plus would come to know how far is the shape from centre of screen.

As this program can be applied to pick bottles, boxes etc as from top the bottles, boxes etc have a 2D

shape. For e.g. bottle top would look like a circle, carton box from top would look like a square or

rectangle etc. So to have higher rate of accuracy, higher speed of work etc we need to have our camera

put just above the gripper of the robotic arm. So, as to be precise about the location of the object.

Plus we have to put the camera in an environment so that there is no noise in the image. Otherwise the

program won’t be able to detect a particular shape as there would be numerous blobs in form of noise.

CONCLUSION:

According to me this program is better than any other program, which just uses bitmap, shape matching

etc is that this program has many constrains in deciding the shape of a object. The object has to undergo

various constrains as to finally make it under a particular shape. Where as in bitmap or shape matching

success isn’t guaranted. The reason being the bitmap shape or the shape matching shape which we

input into the program can vary from the object under the camera. The object or shape should exactly

resemble the shape in the program. If not it won’t be detected.

Doing this assignment I have learnt a lot about robotics. I have come to know how does a robot see and

analyze the information, which we human do with very less effort. I have come to under much about

image processing and motion of robot practically which I had never known earlier. This assignment may

help me a lot when I do my final year project. As it has a wide scope.

43

REFERENCE:

ONLINE

http://www.wisegeek.com/what-is-image-processing.htm accessed on 21st April, 210

http://www.gisdevelopment.net/tutorials/tuman005p.htm accessed on 24th April, 210

http://www.fiserlab.org/manuals/procheck/manual/man_cv.html accessed on 24th April, 210

http://www.endurance-rc.com/roborealm.html accessed on 26th April, 210

http://www.roborealm.com accessed on 27th April, 210

http://www.ukiva.org/pages/applications.html accessed on 27th April, 210

http://www.cognex.com/ApplicationsIndustries/IndustryApps/default.aspx?id=72 accessed on 28th

April, 210

http://www.matrox.com/imaging/en/press/feature/packaging/robot/ accessed on 29th April, 210

http://www.spacetechhalloffame.org/inductees_1994_Digital_Image_Processing.htm accessed on 29th

April, 210

Books

Tinku Acharya and Ajoy K. Ray, Image Processing - Principles and Applications

Wilhelm Burger and Mark J. Burge (2007), Digital Image Processing: An Algorithmic Approach

Using Java

R. Fisher, K Dawson-Howe, A. Fitzgibbon, C. Robertson, E. Trucco (2005), Dictionary of Computer

Vision and Image Processing

Bernd Jahne (2002), Digital Image Processing

Tim Morris (2004), Computer Vision and Image Processing

44