trafficassist - princeton universityalaink/smartdrivingcars/finalpaper-1.pdf · the software has...
TRANSCRIPT
![Page 1: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/1.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
–Independent Work Report Spring, 2013–
TrafficAssist
A monocular camera based approach to identifying
traffic situations
Horia Stefan Radoi, prof. Alain Kornhauser (advisor,ORFE)
Abstract
This paper accompanies a software designed to implement a traffic situation
identificator. It identifies lane markings and other traffic elements (other vehicles, obstacles in
the road etc.), and was designed to be used as part of the visual input of an autonomous vehicle,
in addition to the stereo vision. It manages to identify road situations in real time, and to signal
the driver of possible scenarios. It creates a platform for a robust, platform and vehicle
independent software that requires little to no calibration to identify relevant traffic situations.
1
![Page 2: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/2.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
1. Introduction
Imagine you are out, on a Casual Wednesday, after celebrating your friends’ anniversary. Both
you and your friend had a couple too many drinks, and are in no way capable of driving. You decide to
call a taxi, and you insist that your friend should do the same. He insists that he is fine, and convinces
you that he can drive, and that he can get home safely. He has, after all, driven for the past 5 years, and
knows the road like the back of his hand. The bar you met up is close to a highway, and from the
highway, your friend can get home in 5 minutes. He drives off , and as you are about to enter the taxi,
you hear a loud crash, which you ignore. The following day you find that your friend was involved in an
accident, and he is in serious trouble with the law. You are happy that your friend was not involved in a
fatality, and try to make sense of what happened the previous night.
Unfortunately, the scenario we presented is not unique, and can happen multiple times a day. It
is estimated that, in 2010, in the United States, there were 32,885 deaths and around 2,239,000 injuries
as a result of the 5,419,000 auto vehicle crashes . Out of these fatalities, 31% were alcohol related .1 2
Since it is beyond the scope of this paper to criticize drunk driving, and given the inevitability of such
behaviour, given the current legislation and human mentality, it is up to car manufacturers and engineers
1 http://wwwnrd.nhtsa.dot.gov/Pubs/811630.pdf2 http://www.cdc.gov/motorvehiclesafety/impaired_driving/impaireddrv_factsheet.html
2
![Page 3: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/3.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
to provide “safety nets”, should this behaviour happen, and limit the damages to both the driver and to
the people involved in crashes.
Recent years have seen a multitude of safety devices implemented in auto vehicles, starting from
the mandatory seat belts and airbags for protection, continuing with ABS, Automatic Shifting, Cruise
control and GPS receivers integrated into the car, and ending with assistance for stop and go traffic and
parking assistance. These additions have the role of protecting the driver and aiding him in the driving
activity, by removing mechanical distractions(changing gears, pressing the gas pedal while on the
highway) and providing an audible or visual assistance in the case of slow movement (parking or pulling
out of a parking spot, stop and go traffic etc.).
Perhaps the most important function that the future reserves is that of fully autonomous vehicles:
vehicles that can drive themselves based on information they acquire from their surroundings and from
GPS data, vehicles that can interact with each other, that can cluster and drive together, making better
informed decisions about traffic flow, and, slowly, developing a fully autonomous traffic system, in which
the driver is completely removed from the control loop. This future has the potential of drastically
reducing the number of drunkrelated accidents, by the sheer virtue of removing the concept of a drunk
driver.
TrafficAssist, the software that accompanies this paper aims to help shape such an environment.
It provides a monocular camera assistance system that can be attached to any vehicle’s dashboard, and
it will help the user identify traffic lanes and oncoming traffic. The software is aimed at being integrated
3
![Page 4: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/4.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
into an autonomous vehicle, and provides valuable information about the traffic conditions, information
which can be used together with a stereo vision system to determine exact positions of vehicles and
road obstacles based on the velocity and orientation of the car they are attached to. It is designed to
work using a general, off the shelf webcam for image acquisition, and it is developed as a stand alone
application on a nondedicated laptop. In its final implementation, it will use a similar input configuration,
but run on a dedicated vision processing server integrated into the vehicle.
2. Overview
2.a. Background
The necessity of autonomous vehicles rose as the demand for more safety in auto vehicles
increased. There is currently legislation for more than 70 safety devices that can be and should be
incorporated into an auto vehicle, and all are designed to aid the driver either in driving or in case of a
collision. One characteristic of driving is its repetitiveness, which can become strenuous on a driver,
whose full attention needs to be focused on the road and on his surroundings, most of the time exercised
in an environment which rarely changes (while driving on a highway or Interstate, or in stop and go
traffic). At the same time, the road infrastructure is carefully legislated, and the driving environment has a
strict set of rules for every possible situation. It is a shame, then, that such a repetitive action cannot be
automated, and the human driver (“Errare humanum est” to err is human) be removed from the driving
loop, and allowed the freedom to enjoy his driving experience without him having to worry about other
4
![Page 5: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/5.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
traffic participants.
Previous work has been done in autonomous vehicle engineering. Starting with the DARPA
challenges in 2004, 2005 and 2007, and continuing with the Google Driverless Car , it has been34 56
proven that autonomous vehicles can be built and that they can interact with humandriven traffic and
obey traffic laws. Combined with the emergence of hybrid and electric cars, the field of autonomous
vehicles has a huge potential of growth and is already being implemented, if at a lower level now , at a7
higher level as soon as next year , with Mercedes’ 2014 S Class.8
Teams that have participated in the DARPA challenges have continued their research, and,
including Princeton’s Autonomous Vehicle Engineering
2.b Implementation
The software was designed to be used as a standalone application, or as part of an integrated
system. In its current form, it identifies lanes and has a clear representation of object edges. It will be
able to implement an object classification and object tracking, by grouping the identified elements into
three categories, using the vehicle it is located in as a point of reference: moving in the same direction as
the vehicle (small difference in size, object shrinks as time goes by, or size difference is small when it
3 http://archive.darpa.mil/grandchallenge04/4 http://archive.darpa.mil/grandchallenge/5 http://www.ted.com/talks/sebastian_thrun_google_s_driverless_car.html6 http://orfe.princeton.edu/courses/orf401/handouts/Orf401JoshuaHerbach2013.pdf7 http://www.nytimes.com/2007/10/24/automobiles/autospecial/24park.html?fta=y&_r=08 http://www.autonews.com/article/20130228/BLOG06/130229874
5
![Page 6: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/6.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
brakes), stationary objects in the scenery, or objects not moving relative to the vehicle (they appear as
growing at a constant rate that is consistent to the measured speed of the vehicle) and objects moving
towards the vehicle (size growing very fast, and rate of change consistent with a speed greater than that
measured by the test vehicle).
The software has the capability to be integrated into a larger system, as a supplementary visual
input method for an autonomous vehicle, in addition to its stereo vision sensors, and can better interpret
data if it has access to an accelerometer (to measure the real speed of the test vehicle).
While GPS positioning would be a nice addition to the project, it is not necessary, since the
software can be programmed to identify an environment based on visual cues (ie. many wide lanes for a
highway environment, houses or buildings for an urban environment etc.)
3. TrafficAssist
3.a Hardware
This project was developed and tested on a Sony VPCF127FX laptop computer(Intel®
Core™ i7 CPU Q740 @ 1.73 GHz x 8) , running Ubuntu 12.04, 64bit.
We have decided on a top of the shelf webcam ( Logitech C920, 1080p widescreen). This was
consistent with our wish to make the hardware requirement as accessible as possible, and, at the same
time, took advantage of this webcam’s high pixel count (possibility to capture highlydetailed images for
the puppet Matte and the background) and autofocus. While, at times, the constant autofocus proved
6
![Page 7: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/7.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
to be a nuisance in image acquisition, the algorithms used were able to account for this flaw.
For the purposes of testing this software, we have also used a 2004 BMW Z4, and acquired
images in the area surrounding Princeton University, on Route 1 and Washington Road. The camera
was placed on the passenger’s side, close to the center of the vehicle, at the dashboard level. It was
physically placed at 1.5 meters from the front of the car, at a height of 1 meter from the road level.
3.b Software OpenCV and Python
OpenCV is an open source vision platform, released under a BSD licence, free for both
academic and commercial use. It was written in C and C++, and provides interfaces for C, C++,
Python and Java, under Windows, Linux, Mac OS, iOS and Android. OpenCV uses the full
computational power of a multicore computer, and provides a good framework for realtime image
processing.
OpenCV contains over 500 optimized functions, for immediate usage. It features a growing
online community, and very well documented community support, through both their own wiki and the
StackOverflow website. This availability of algorithm implementation made it an essential tool for this
project. The learning curve is steep, but once one understands the basic functionality, OpenCV proves
to be a powerful image manipulation software. The OpenCV libraries and functions are built in a way
that enables interaction between different programmers, by providing cross platform and language
independent support. OpenCV features tutorials for multiple platforms, and the documentation files
7
![Page 8: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/8.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
feature all available programming implementations (C++, C, Python etc.).
The OpenCV version used in this project is OpenCV 2.4.2 .
This project was coded in Python. It is a very lightweight programming platform, providing the
power of C++ and enabling the same objectoriented programming functionality, while reducing the
amount of code needed to be written. It provided a very good educational environment for the initial
implementation of the OpenCV functions. Unfortunately, since the OpenCV developers are still
implementing functions in Python, the lack of documentation for those functions proved to be an
obstacle in implementing the program.
The Python build used in the early phases of project was 2.7.3.
3.c Implementation
The software is designed to use both realtime input and prerecorded input. For consistency,
during testing, a prerecorded video sequence was used as input.
Once a stream is determined, the software processes each frame individually, by first
determining the edges of the objects present (Using Sobel Operators and Canny edge detectors). This
information is going to be used by the object classifier.
The Canny edge detector is used with Hough transformations to determine the straight lines in
the image. These lines, resulting from the prominent edges, are the basis of the lane identification of the
software.
8
![Page 9: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/9.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
Future development includes identifying objects into the three classes described earlier (moving
with, away from and towards the camera) and extrapolating the POV image to a topdown (bird’s eye)
view of the world. By providing this extrapolation, a future autonomous vehicle can identify the best
route to avoid road congestion or to tackle traffic problems as they arise.
4. Tests and results
The system was tested in a highway setting around Princeton, NJ.
The videos captured identify common traffic situations, such as stop and go traffic, following a
vehicle, following a vehicle that brakes and accelerates. A test video was captured in an urban
environment as well (Washington Road), and another test video captured a curving road (exiting a
turnpike).
So far, the results are satisfying, with the sobel operator identifying contours and creating a good
basis for lane detection. Future developments include restricting the region of interest of the image to
focus only on the road ahead (thus ignoring what happens to the sky and on the reflection of the car),
and a better identification of discontinuous lines.
Some test results can be seen in the Appendix.
9
![Page 10: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/10.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
5. Related Work
5.a PAVE
In 2005 and 2007, Princeton’s Autonomous Vehicle Engineering team participated in the two
DARPA challenges. Both teams led by professor Alain Kornhauser, composed of undergraduate
students, built two autonomous vehicles to participate in each respective competition. Even though the
cars were functional, some of the algorithms used in their development became obsolete, as did some of
the hardware on the vehicles.
In 2012, a group of PAVE students including myself started porting the software of PAVE from
the Microsoft Robotics Studio to the Robot Operating System (ROS), which has a better support for
OpenCV implementations. The hardware used on these vehicles included both color stereo vision
cameras and monocular, black and white cameras. The approach that PAVE takes is that since humans
can drive using only vision, it is not unreasonable to consider a selfdriving car that utilizes only vision
porcessing.
The end goal is that the TrafficAssist software is going to be used on the PAVE vehicle.
5.b Google Self Driving car
The Google Self Driving car is perhaps one of the best known examples of autonomous
vehicles. The Google project, started under the coordination of Sebastian Thrun (the coordinator of the
Stanford teams at the 2004, 2005 and 2007 DARPA challenges), has been leading both technological
10
![Page 11: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/11.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
advances and legislation to be pushed towards a future of autonomous driving vehicles.
In a presentation given earlier this semester, during ORF 401, Joshua Herbach, a former
Princeton Undergraduate and member of the PAVE project, currently working on the Google Self
Driving car, showed some of the mechanics of the Google vehicle. While different from the PAVE
approach in that most of the information regarding the environment comes from the huge LIDAR unit
located above the vehicle (to obtain a representation of its surroundings), the Google car uses vision as
a means of verification of its input data. This approach takes into account the huge database that Google
has at hand (with the Google Street view), and uses a group of people to map out the specifics of each
environment (traffic signs, speed limits etc.). In addition, the Google approach is sensitive to change, in
that, if a car identifies that an environment is not consistent with its databases, it prompts its driver to
take control of the car, until the area is updated.
While Google’s approach is working, it is severely ignoring the potential of vision processing:
modifications in the environment happen all the time (branches falling, potholes forming, water on the
road etc.), and a vision based approach can easily identify these obstacles and act accordingly.
6. Limitations and further work
The current limitations of the project come from the restricted time frame given by the University
in completing the independent work. While the project is not currently in a finished state, it will still be
improved and more features can be added.
11
![Page 12: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/12.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
The current lane detection system assumes continuous lane markings, and does not take into
account perspective. So far, it has only been tested from one point of view, but we plan to extend it to
be independent of location (perhaps needing only some calibration once it is mounted into a new place).
Some alternative algorithms or approaches to implemented algorithms are available, and they
are presented as more efficient and more exact approximations of lane detections or object
identification. A quick means of efficiency is to restrict the ROI of the image to only an area located9
underneath the horizon line (since that is mostly where the road will be). Alternatively, ffthe image can be
classified into color regions, and those corresponding to “sky” or “side of the road” can be ignored (for
the purposes of lane detection and traffic identification).
7. Conclusions
While this project is still in an early state, we are working on developing it into a fully functional,
standalone project. Throughout this project we have identified the potential of monocular cameras in
image processing, and have formed a basis of knowledge as to how a series of 2D images can be used
to simulate a 3D environment.
Moreover, starting at the 2D PointofView perspective, the software has shown a potential of
extrapolating to a 2D, bird’s eye perspective, thus aiding in programming an Artificial intelligence
module to determine the best path it can take to reach its destination.
9 Kim, ZuWhan. "Robust lane detection and tracking in challenging scenarios." Intelligent TransportationSystems, IEEE Transactions on9.1 (2008): 1626.
12
![Page 13: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/13.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
8. Appendix
Some of the images resulted from testing the system:
Fig 1.a Test frame of a highway traffic situation
Fig 1.b Canny edge detection and Hough line transformation used on the previous image to detect
lanes. Some false positives are identified in the reflection of the car and on the sky, but these issues can
be resolved by restricting the ROI of the image to only the road surface (or anything below the horizon
line).
13
![Page 14: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/14.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
Fig 1.c Two successive Sobel operators applied to the same image, once in a vertical direction and
once in a horizontal direction, are here superimposed to create a good representation of the edges of the
objects. Individually, the Sobel operators can provide useful information about the surroundings (the
horizontal one providing a good approximation of where the shadows of the vehicles are, and the
vertical one providing another means to identify traffic lanes)
Honor Pledge
This paper represents my own work, in accordance with University Policies.
[Horia Stefan Radoi]
14
![Page 15: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/15.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
Acknowledgements
I would like to thank my independent work advisor, professor Alain Kornhauser, for
coordinating my work and steering me in the right track, even though I was a pain to work with. I would
also like to thank the Computer Science department at Princeton University, along with the IW
coordinators, for always reminding me of deadlines and for all the passion and dedication they have in
teaching me, us to become better scholars.
I would also want to thank all the people who have supported me in conducting my reasearch,
Kevin Yang for allowing me to use his car for weird purposes and all my friends at Colonial Club who
have supported me in their own ways.
Lastly, I would want to reiterate my gratitude to professor Adam Finkelstein for allowing me to
learn OpenCV and develop as a vision processing programmer.
References and bibliography
Academic papers
Kim, ZuWhan. "Robust lane detection and tracking in challenging scenarios." Intelligent
Transportation Systems, IEEE Transactions on9.1 (2008): 1626.
Wang, Yue, Eam Khwang Teoh, and Dinggang Shen. "Lane detection and tracking using
BSnake." Image and Vision computing 22.4 (2004): 269280.
15
![Page 16: TrafficAssist - Princeton Universityalaink/SmartDrivingCars/FinalPaper-1.pdf · The software has the capability to be integrated into a larger system, as a supplementary visual input](https://reader033.vdocuments.net/reader033/viewer/2022042101/5e7dfe8f7cc4404c9964b8dd/html5/thumbnails/16.jpg)
Horia Stefan Radoi
Spring 2013 Independent Work
Wang, Yue, Dinggang Shen, and Eam Khwang Teoh. "Lane detection using spline model."
Pattern Recognition Letters 21.8 (2000): 677689.
Wang, Yue, Dinggang Shen, and Eam Khwang Teoh. "Lane detection using catmullrom
spline." IEEE International Conference on Intelligent Vehicles. 1998.
OpenCV, Python Tutorials Tutorials:
Rahman, Abid, OpenCV Python Tutorials, <opencvpython.blogspot.ro>
Rahmbihia, Jay, Paranoid Android, <http://www.jayrambhia.com/blog/>
<http://stackoverflow.com>:
http://stackoverflow.com/questions/tagged/opencv
http://stackoverflow.com/questions/tagged/opencv+python
<http://docs.opencv.org/>
<http://www.aishack.in/>
Pisarevsky, Vadim. "Introduction to OpenCV." Intel Corporation, Software and Solution
(2007).
Szeliski, Richard. Computer vision: Algorithms and applications. Springer, 2010.
16