artificial vision in road vehicles

Artificial Vision in Road Vehicles

By: MASSIMO BERTOZZI, ALBERTO BROGGI, MASSIMO CELLARIO, ALESSANDRA FASCIOLI, PAOLO LOMBARDI, AND MARCO PORTA

Presented by: Ali Agha

April 6th, 2009

Main Motivation

Help the driver in case of failure, for example, due to a lack of concentration or due to drowsiness.

Active Sensors in ITS Example: Laser-based sensors and millimeter-

wave radars Drawbacks

Low spatial resolution Slow scanning speed and Expensive Reflection problems maximum signal level must comply with some safety rules interference among sensors of the same type

Advantages: mm-wave radars are robust to rain and fog measure some quantities, such as movement, in a more

direct way require less performing computing resources, as they

acquire a considerably lower amount of data.

Passive Sensors in ITS

Example: Vision-based sensors Advantages:

Noninvasive Useful in specific applications (lane localization,

traffic signs recognition, Obstacle identification)

Drawbacks: Not robust in foggy, night, or direct sunshine

conditions.

Consideration in design a vision system for ITS applications ITS systems require faster processing than other

applications, since vehicle speed is bounded by the processing rate.

computing engines cannot be based on expensive processors.

no assumptions can be made on scene illumination or contrast and the process should be robust to environmental conditions, such as

sun, rain, or fog and their dynamic changes such as transitions between sun and shadow, or the entrance or exit from a tunnel.

robustness to vehicle’s movements and handling the drifts in the camera’s calibration

Road Following

Road Following

Lane Detection Obstacle Detection Pedestrian Detection

Lane Detection Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth

Road ShapePrior Knowledgeon surface/shape

Two basic problem in Lane Detection are:

1)The presence of shadows

2)Occlusion, caused by other vehicles

Focus of attention

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


Due to both physical and continuity constraints, the processing of the whole image can be replaced by the analysis of specific regions of interest only (the so-called focus of attention), in which the features of interest are more likely to be found.

Focus of attention

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth

Road ShapePrior Knowledgeon surface/shape J. Goldbeck, et al. (1998)

R. Chapuis, et al. (2001)

employs a model both for the road and the vehicle’s dynamic to determine the road portionwhere it is most likely to find lane markings

Focus of attention

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth

Road ShapePrior Knowledgeon surface/shape J. Goldbeck, et al. (1998)

R. Chapuis, et al. (2001)

Dynamically determination of WOI by means of statistical methods; According to the current state and previously detected WOIs.

Fixed Lane Width

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


The assumption of a fixed or smoothly varying lane width allows the enhancement of the search criterion, limiting the search to almost parallel lane markings.

Fixed Lane Width

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth

Road ShapePrior Knowledgeon surface/shape K. Kim, et al. (1995)

D. Pomerleau, et al. (1996)

lane markings can be detected using both neural networks and simple vision algorithms:

two parallel stripes of the acquired image are selected and filtered using Gaussian masks and zero crossing to find vertical edges. The result is matched against a given model.

Fixed Lane Width

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth

Road ShapePrior Knowledgeon surface/shape K. Kim, et al. (1995)

D. Pomerleau, et al. (1996)

based on the processingof the image portion

The determination of the curvature is carried out according to a number of possible curvature models

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


The reconstruction of road geometry can be simplified by assumptions on its shape.

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


Lützeler, et al. & Franke, et al. (1998) & Goldbeck, et al. (1999) K. A. Redmill, et al. (2001) S. L. Michael, et al. (1997) K. A. Redmill, et al. (1997) F. Chausse, et al. (2000) J. Goldbeck, et al. (1999) R. Risack, et al. (1998) S. M.Wong, et al. (1999) X. Youchun, et al. (2000) A. Broggi, et al. (1995) & S. Denasi, et al. (1994)

lane markings are modeled as clothoids. In a clothoid thecurvature depends linearly on the curvilinear reference. This model has the advantage that the knowledge of two parameters only allows the full localization of lane markings and the computation of other parameters

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



dynamic programming optimization method is used to chose among center-line candidates representing the actual geometry of the road

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



uses a polynomial representation forlane markings. lane markings are modeled as parabolas and a simplified Hough transform is used to accomplish the fittingprocedure.

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



relies on a polynomial curve. It assumes a flat road with either continuous or dashed bright lane markings. The history of previously located lane markings is used to determine the region of interest

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



exploits a polynomial road modelization to calculate the impact distance from the vehicle to the nearest road side by considering the intersection between the straight line trajectory followed in case of adriver loss of control and the polynomial function describing the road side

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



proposed to use concentric circles to represent lane boundaries. circular shape models can in fact be better choices than polynomial approximations

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



adopt a more generic model for the road. uses a contour-based method. Actually, only straight or small curved roads without intersections are included in this model. The road model is used to follow contours formed by pixels that feature a significant gradient direction value.

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



uses an edge linking process based on Contour Chains and Causal Neighborhood Windows(areas of interest connected to edge elements). After an initial segmentation phase, the longest chains with slope angles close to 45 and 135 degrees are searched.

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



based on a linear lane model, where road markers are reconstructed as sequences of straight lines.

Road Shape

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth



A generic triangular road model is proposed

A priori knowledge on the road surface/slope

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


The knowledge of the specific camera calibration together with the assumption of an a priori knowledge on the road (i.e., a flat road without bumps) can be exploited to ease the localization of features and/or to simplify the mapping between image pixels and their correspondent world coordinates.

A priori knowledge on the road surface/slope

Road Following

PedestrianDetection

Obstacle Detection

Lane Detection

Focus ofattention

Fixed Lanewidth


M. Bertozzi, et al. (1998)

The GOLD system exploit the assumption of a flat road in front of the vehicle. The lane markings detection is performed in a different image domain, representing a bird’s eye view of the road, which can be obtained thank to the flat road assumption.

Lane Detection (Summery)

Pedestrian Detection

Road Following

Lane Detection

Obstacle Detection


Segmentation with motion

SegmentationWith stereo

Focus ofattention

Recognition ofHuman gait

Recognition ofHuman shape

Segmentation with motionRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



Pros:Use temporal information and is reliable

Cons:Does not detect standing pedestrianNeeds sequence of a few framesJust for detecting moving objects not their

velocity


Lane Detection

Obstacle Detection




Focus ofattention



R. Polana et al. (1995) S. McKenna et al. (1997) R. Cutler et al. (2000)

motion detection with optical flow: They analyzes the scene with a discrete cube, where they assign to each region its average optical flow.


Lane Detection

Obstacle Detection




Focus ofattention




uses a zero-crossing detection algorithm using the convolution of a spatio-temporal Gaussian


Lane Detection

Obstacle Detection




Focus ofattention




uses a subtraction between an image at time t and a version of the same image stabilized with respect to image at instant t-taw.

Segmentation with StereoRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



D. Beymer et al. (1999) L. Zhao et al. (2000)

In surveillance applications, stereo analysis is sometimes used as a cue to build a disparity map of the background for use with background subtraction.

Segmentation with StereoRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



D. Beymer et al. (1999) L. Zhao et al. (2000)

Range thresholding based on stereo analysis for segmentation

Focus of AttentionRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



A. Broggi et al. (2000) C. Curio et al. (2000)

salient regions in opportune feature maps are interpreted as candidates for pedestrians. In the GOLD system, vertical symmetries are associated with candidates for standing pedestrians

Focus of AttentionRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



A. Broggi et al. (2000) C. Curio et al. (2000)

The focus of attention is directed by a composition of 1) a map of the local image entropy, -2) a model-matching module with the shape of a representing human legs, and -3) finally inverse perspective mapping (binocular vision) for the short distance field.

Recognition of Human GaitRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



C. Wohler et al. (2000) R. Cutler et al. (2000) C. Curio et al. (2000)

the ATDNN performs a local spatio-temporal processing to detect the typical pattern of the movement


Lane Detection

Obstacle Detection




Focus ofattention




Periodicity of the human gait is often recognized with traditional methods like the Fourier transform


Lane Detection

Obstacle Detection




Focus ofattention




The periodic movement detected is correlated to an experimental curve derived from the statistical average of human gait periods

Recognition of Human ShapeRoad Following

Lane Detection

Obstacle Detection




Focus ofattention



D. Beymar et al. (1999) & A. Broggi et al. (2000) H. Fujiyoshi et al. (2000) D. Gavrila et al. (2000) V. Philomin et al. (2000) C. Papageorgiou et al. (1999) & A. Mohan et al. (2001)

employ a model for the head and shoulders. This approach is very sensible to scale variation, so multiple models of different scales are needed.


Lane Detection

Obstacle Detection




Focus ofattention




uses a skeletonization procedure.calculate first the centroid of the area and then the distances from the centroid to each border points.


Lane Detection

Obstacle Detection




Focus ofattention




generic forms are tried first, and similar and more detailed shapes afterwards.


Lane Detection

Obstacle Detection




Focus ofattention




Pedestrian Detection (Summery)

Thank you

Questions?

New developments in cameras

current cameras include new important features that permit the solution of some basic problems directly at sensor level. For example, image stabilization can be performed during acquisition, while the extension of camera dynamics allows one to avoid the processing required to adapt the acquisition parameters to specific light conditions. The resolution of the sensors has been drastically enhanced, and, in order to decrease the acquisition and transfer time, new technological solutions can be found in CMOS sensors, such as the possibility of dealing with pixels independently as in traditional memories. Another key advantage of CMOSbased sensors is that their integration on the processing chip seems to be straightforward.

Choosing appropriate camera Many different parameters must be evaluated for the design and choice of an image acquisition device. First of all, some parameters tightly coupled with the algorithms regard the choice of monocular versus binocular (stereo) vision and the sensors’ angle of view (some systems adopt a multicamera approach by using more than one camera with different viewing angles, e.g., fish eye or zoom). The resolution and the depth (number of bit/pixel) of the images have to be selected as well (this also includes the selection of color versus monochrome images).

Other parameters—intrinsic to the sensor—must be considered. Although the frame rate is generally fixed for CCDbased devices (25 or 30 Hz), the dynamics of the sensor is of basic importance: conventional cameras allow an intensity contrast of 500:1 within the same image frame, while most ITS applications require a 10 000:1 dynamic range for each frame and 100 000:1 for a short image sequence. Different approaches have been studied to meet this requirement, ranging from the use of CMOS-based cameras with a logarithmically compressed dynamic [6], [7] to the interpolation and superimposition regarding values of two subsequent images taken from the same camera [8].

Obstacle Detection (Vehicle Detection) M. Lutzeler et al. (1998) S. Kyo et al. (1999) S. Denasi et al. (2001)

W. Kruger et al. (1995) S. M. Smith et al. (1995) Z. Hu et al. (2000) F. Marmoiton et al. (1995)

Obstacle Detection (Generic Obstacle) – Single camera

U. Franke et al. (1998) M. Bertozzi et al. (1998) H. Hattori et al. (2000) M. Hariyama et al. (2000) Y. Ruicheck et al. (2000) D. Koller et al. (1995) S. Denasi et al. (1994) M. Leeuwen et al. (2000) C. Knoeppel et al. (2000)

Obstacle Detection (Generic Obstacle) – Multiple cameras

artificial vision in road vehicles

Documents

fixed lane widthk

parallel lane markings

fixed lane widththe

varying lane width

road vehiclesby

road shapeltzeler

road portionwhere

visionbased sensorsadvantages