machine vision robotic approach

7/31/2019 Machine Vision Robotic Approach

1/42

The automatic acquisition and analysis of images to obtain desired data for

controlling a specific activity.

A guidance system that gives a robot the ability to see what it is doing and react, as

a human would, to changes in positioning.

Systems that use video cameras, robots or other devices, and computers to visually

analyze an operation or activity. Typical uses include automated inspection, optical

character recognition and other non-contact applications.

Machine vision is the application of computer vision to factory automation. Just as

human inspectors working on assembly lines visually inspect parts to judge the

quality of workmanship, so machine vision systems use digital cameras and image

processing software to perform similar inspections. A machine vision system is a

computer that makes decisions based on the analysis of digital images.


2/42

Vision is the Most Powerful Sense for Humans

It provides an enormous amount of information about the environment and

enables analysis for rich, intelligent interaction in dynamic environment


3/42

Sensors

Vision Based

Sensors

ProprioceptiveExteroceptive

Passive Active

Thermal

Sensors

Vision Based

Sensors

Laser Range

Finder

Eye

Scanner(Laser)

Sonar

ScannerCCD CMOS


4/42

CCD (Charge coupled Device) is a device with light-Sensitive photo cells which is

used to create bitmap images.

Most popular basic ingredient of robotics vision systems Today.

Is an array of light-sensitive picture elements, or pixels, Usually with between

20,000 and several million pixels total.

The basic light measuring is colorless : it is just measuring the total number of

photons that strike each pixel in the integration period.

CCDs

Three ChipsSingle Chip


5/42


6/42

The pixels on the CCD chip are grouped into 2x2 sets of four, then Red, Green, and

Blue Dyes can be applied to a color filter so that each individual pixel receives only

light of one color.

Normally 2 pixels measure Green while one pixel

each measures Red and Blue light intensity (GRGB).

In the other hand it can be as a single chip RGBE format

which uses emerald (cyan blue) instead of green for measuring .

NOTE: The Number of pixels in the system has been effectively cut

by a factor of four.


7/42

Splits the incoming light into three complete (lower intensity) copies. Three

separate CCD chips receive the light, with one Red, Green, or Blue filter our each

entire chip.

Silicon Absorbs different wave-lengths of light at different depths.

Three Chip CCD can capture red, Green and Blue light at every pixel location

As a result ofOne Chip CCD mosaic sensors, it just capture 50% of the green and

only 25% of the red and blue light


8/42

The complementary Metal Oxide Semiconductor chip is a significant departure

from the CCD

In most CMOS devices, there are several transistors at each pixel that amplify and

move the charge using more traditional wires.


9/42

The CMOS approach is more flexible because each pixel can be read individually.

CCD sensors, as mentioned above, create high-quality, low-noise images. CMOS

sensors, traditionally, are more susceptible to noise.

Because each pixel on a CMOS sensor has several transistors located next to it, the

light sensitivity of a CMOS chip tends to be lower. Many of the photons hitting the

chip hit the transistors instead of the photodiode.

CMOS traditionally consumes little power. Implementing a sensor in CMOS yields alow-power sensor.


10/42

CMOS chips can be fabricated on just about any standard silicon production line, so

they tend to be extremely inexpensive compared to CCD sensors.

CCD sensors have been mass produced for a longer period of time, so they aremore mature. They tend to have higher quality and more pixels.


11/42

The key Disadvantages of CCD and CMOS cameras are primarily in the areas of

inconstancy and dynamic range.

The Second Class of Disadvantages relates to the behavior of a CCD chip in

environment with extreme Illuminations.


12/42

Eye Scanner Sensor uses a single beam laser for modeling the environment


13/42

Collects Data similar the way a sonar range finder sensor does

Uses Vision based Methods for data extraction and analysis in making 3D maps and

localizing the robot such as 2D evidence grid.


14/42

DEPTHX was the first research robot used this technique for exploring Zacaton

sinkhole in central Mexico (Diameter:110m Depth: over 350m)


15/42

Advantages :

Lots of data can be extracted from the Image

Faster in some cases

Disadvantages:

Is not Accurate in all environments

Because it is passive (CCD and CMOS) type environment has more effect onanalysis

Needs much more powerful processors (microcontrollers cannot be used in most

cases)


16/42

Line Follower Robot with Photocell sensor Self Drive Volkswagen Golf GTi


17/42

Range Sensing is extremely important in mobile robotics as it is a basic input for

successful obstacle avoidance

A number of sensors are popular in robotics explicitly for their ability to recover

depth estimates: ultrasonic, laser range finder, optical range finder. So it is natural

to attempt to implement ranging functionality using Vision chips (CCD,CMOS,) aswell.

A fundamental problem with visual images makes range finding relatively difficult.

Any vision chip collapses the 3D world into the 2D image plane, thereby losing

depth information.

If one strong assumptions regarding the size of objects in the world, or their

particular color or reflectance, then one can directly interpret the appearance of

the 2D image to recover depth. But such an assumption are rarely possible in real-

world robot applications.


18/42

The general solution is to recover depth by looking at the several images of the

scene to gain information.

The images used must be different, so that taken together they provide additional

information

An alternative is to create different images by changing the view point , or by

changing the camera geometry such as the focus position.

Depth from Focus - Depth from Defocus is the Basic Concept for Visual range

finding.


19/42

Relies on the fact that image properties change as a function of camera parameters

If the image plane is located at distance e from the lens, all light will be focused at

single point on the image plane and the object voxel will be focused.

When the image plane is not at e, then the light from the object voxel will be caston the image plane as a blur circle, and the radius R of the circle can be

characterized according to the equation.

e

LR

2

edf

111


20/42

The distance to near objects can therefore

be measured more accurately than that

to distant objects, just as depth from

Focus techniques.

The accuracy of the depth estimate increases

With increasing baseline b

As bis increased, because the physical

Separation in cameras is increased,

Some objects may appear in one camera butNot in the other. Such objects will not

Be ranged successfully.


21/42

LearningPhase Test


22/42


23/42

An important aspect of vision based sensing is that the vision chip can provide

sensing modalities and cues that no other mobile sensor provides.

One such novel sensing modality is detecting and tracking color in the

environment.

Advantages of Color Detection:

detection of color is straightforward function of a single image

because color sensing provides a new, independent environmental cue, if it iscombined with existing cues, such as data from stereo vision or laser range finding,

we can expect significant information gains.


24/42

Color Tracking Camera for Robots


25/42

Using Vision chip as Color Tracking Sensor for line following


26/42

Using Vision chip as Color Tracking Sensor for line following with collision avoidance


27/42

An Autonomous mobile robot must be able to determine its relationship to the

environment by making measurements with its sensors and then using those

measured signals.

Vision-based feature extraction can affect a significant computational cost,

particularly in robots where the vision sensor processing is performed by one therobots main processors

The method must operate in Real-Time. Mobile robots move through the

environment and so the processing simply cannot be an off-line operation.

The Method be robust to the real world conditions outside of the laboratory. This

means that carefully controlled illumination assumptions and carefully painted

objects are unacceptable requirements.


28/42

Vision-Based interpretation is primarily about the challenge of reducing

information.

A sonar unit produces perhaps 50 bits of information per second. By contrast, a

CCD camera can output 240 million bits per second.

Image Preprocessing: it is important to note that all vision-based sensors supply

images with such significant amount of noise that a first step usually consist of

cleaning the image before lunching any feature extraction algorithm.


29/42

A 3x3 kernel table weighted as:

Such a Low-pass filter effectively removes high-frequency noise, and this in turn

causes far more stable.

121242

121

16

1

G


30/42

Without Gaussian Smoothing With Gaussian Smoothing


31/42

Using Vision chip (Infrared CCD) for extracting specific objects base on color


32/42

The single most popular local feature extractor used by the mobile robotics

community is the edge detector.

Edges define regions in the image plane where a significant change in the image

brightness takes place.

Edge detection significantly reduces the amount of information in an image, and is

therefore a useful potential feature during image interpretation.

Optimal Edge Detection Canny. The current reference edge detector through out

the vision community was invented by john Canny in 1983


33/42

The very first step is preprocessing. Each input image is gray scaled and contour-

filtered using the Canny edge

detector

Using exhaustive scanning. In anX Y image with an N M template, we first try to

match the windowdefined by the rectangle (0, 0,N,M);

after that the one defined by (1, 0,N + 1,M), and so on until reaching

the end of the image at that scale.

Using random sampling. In an X Y image with an

N M template, we select a fixed number of samples

proportional to the size of the image. This scanning

method accelerates the process with a sacrifice in

Precision.


34/42

In the offline experiments we use exhaustive scanning because runtime performance is

not an issue. The online version also uses exhaustive scanning. However, note that

the online version could be made faster by using random

sampling, but in this case not all positions in the image will

be scanned in each frame.

More templates means a better definition of the

class of interest but also translates into a slower

matching process. The templates are taken from

photographs of the object of interest after contour

filtering it and obtaining the relevant connected

components.


35/42

The System Was Tested with an

ActivMedia Robotics Pioneer 2 mobile

robot. The online version (onboard the

robot) uses the randomized scanning

method previouslydescribed.


36/42


37/42

They serve compact representations of the entire local region.

Direct Extraction: image Histograms

in still mode, with rotation of the robot or camera pixel positions will change,

although the new image will simply be a rotation of the original image. But we

intend to extract image feature via histogramming. because histogramming is a

function of a set of pixel values and not the position of each pixel the process is

pixel position invariant .


38/42

Unwarped Planner reconstruction of partial viewOriginal Omni Directional Image


39/42


40/42

The DARPA Grand Challenge is a United States

government-sponsored competition that aims

to create the first fully autonomous vehicles

capable of competing on an under-300 mile,

off-road course in the. This annual challenge

took place for the first time on March 13, 2004and was sponsored by the Defense Advanced

Research Projects Agency of the U.S.

Department of Defense.


41/42


42/42

Autonomous Mobile Robots (Siegwart Nourbakhsh, MIT Press)

Artificial intelligence illuminated (Coppin, Jones and Bartlett Publishers)

DORPA Grand Challenge Documents

Alice: An Information-Rich Autonomous Vehicle for High-Speed Desert Navigation

(California institute of technology)Vision-Based Control of Mobile Robot (John Hopkins University)

A Study of CMOS Cameras (Auburn University)

A system for Vision-Based Human Robot Interaction (Orebro University)

A Method to Detect Victims in Search and Rescue Operations using Template

Matching (Simon Bolivar University)

Real-Time Exploration in underwater tunnels (Carnegie Mellon University)