research article development of a real-time detection...

Research ArticleDevelopment of a Real-Time Detection System forAugmented Reality Driving

Kuei-Shu Hsu,1 Chia-Sui Wang,2 Jinn-Feng Jiang,3 and Hung-Yuan Wei3

1Department of Applied Geoinformatics, Chia Nan University of Pharmacy & Science, No. 60, Section 1, Erren Road,Rende District, Tainan 71710, Taiwan2Department of Information Management, Chia Nan University of Pharmacy & Science, Tainan 717, Taiwan3Project Coordination and Liaison Section, Planning and Promotion Department, Metal Industries Research and Development Center,Taipei, Taiwan

Correspondence should be addressed to Kuei-Shu Hsu; [email protected]

Received 18 September 2014; Accepted 7 December 2014

Academic Editor: Mo Li

Copyright © 2015 Kuei-Shu Hsu et al. This is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Augmented reality technology is applied so that driving tests may be performed in various environments using a virtual realityscenariowith the ultimate goal of improving visual and interactive effects of simulated drivers. Environmental conditions simulatinga real scenario are created using an augmented reality structure, which guarantees the test taker’s security since they are not subjectto real-life elements and dangers. Furthermore, the accuracy of tests conducted through virtual reality is not influenced by eitherenvironmental or human factors. Driver posture is captured in real time using Kinect’s depth perception function and then appliedto driving simulation effects that are emulated by Unity3D’s gaming technology. Subsequently, different driving models may becollected through different drivers. In this research, nearly true and realistic street environments are simulated to evaluate driverbehavior. A variety of different visual effects are easily available to effectively reduce error rates, thereby significantly improvingtest security as well as the reliability and reality of this project. Different situation designs are simulated and evaluated to increasedevelopment efficiency and build more security verification test platforms using such technology in conjunction with driving tests,vehicle fittings, environmental factors, and so forth.

1. Introduction

The application of virtual reality to games that are availableon computers or mobile phones is currently a very populartopic. Multiple cameras may be utilized to observe a user asthat user simulates driving a vehicle. Moreover, to analyzethe conditions of operating a vehicle, situation simulations(in addition to environmental factors and external variableconditions) may be employed to test the user’s response,action changes, and so forth. Such simulations may even bea criterion for developing vehicle equipment and parts. Userhabits are obtainable through the direct operation of virtualreality.

The subject of this study is to apply a stereo visionsystem to a virtual reality system. A computer’s vision systemmay be divided roughly into plane vision and stereo vision,which are different in their capabilities to estimate the depth

information of target objects in an image (also referred to as“depth perception”); this division makes a computer visionsystem more practical. Some of its many practical applica-tions include remote sensing and monitoring, medical imageprocessing, robot vision systems, military reconnaissance,mineral exploration, and cartography.

In recent years, computer vision systems have madebig advances in both the theory and algorithms of imageprocessing as well as in practical applications. For example, a3D stereo object in spacemay be captured as a 2Dplane imageby a charge-coupled device camera through image processingmethods—such as image transforms, image enhancement,thresholding, filtering, edge detection, and thinning [1, 2].That object may then be used as image information for appli-cations of feature extraction, recognition, visual servicing,and so forth [2–4]. Furthermore, advanced techniques haveled to the development of 3D stereo vision systems. However,

Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2015, Article ID 913408, 5 pageshttp://dx.doi.org/10.1155/2015/913408

2 Mathematical Problems in Engineering

Driverless car Visual reality

Scripting

Services

3D models

Vehicles

Particle systems

PeopleDate sync.

Intelligent and energyefficiency

System integration

Warning system

Figure 1: Research flowchart.

there are still many core technologies to be achieved beforecomputer vision can be compared to human vision [5, 6].

To perform tests in different environments using virtualreality scenarios and create a realistic feel, environmentalconditions that simulate real scenarios must be created.Therefore, we employ Unity software to create virtual realitysystems and then use those systems in conjunction with vari-ous scenarios. Our systems may subsequently be promotedfor use to smart phone platforms. Using virtual reality forsimulations is not only safe but also convenient. Furthermore,testing accuracy is not affected by environmental factors thatmay be induced by tests in the real world.

2. Research Method

This research uses streetscape, which is commonly used indriving, as the scenario. SketchUp software is used to modelandmap an entire city. After the model is drawn, it is import-ed to Unity to create a virtual reality interface. Other repre-sentative scenarios for testing will be created after discussionof this research to increase the accuracy and sufficiency ofmultidimensional data collection.

Figure 1 shows a picture an “augmented reality driver testsystem” is constructed.The functions of this platform includeboth virtual reality and augmented reality, making it oneof the most complex human-computer cooperation machinesystems. In the system, the interactive motion simulatorallows the behavioral model of the driver to enter the virtualscenario and operate the vehicle with a steering wheel. Activeand augmented reality technologies are integrated, and twodifferent interactivemodels of augmented reality interactionsare proposed to invite the driver to perform a practicalitytest for different emergency conditions. Several streetscape3D models are established for the project.

Therefore, the project aims to provide unique, prior-to-development test operations for a driver behavior test. First,the driver is allowed to perform driving behavior analysis inan emulated environment for driver behavior testing thatmayalso be used as a test platform for vehicle products. Second,

Table 1: Abstract table of illustrative statistics for different back-ground variable items.

Backgroundvariable item Group Number Percentage

Gender Male 100 50.0%Female 100 50.0%

Major usepurpose

Goal 30 30.0%Entertainment 37 37.0%Development 33 33.0%

virtual reality is used to derive new teaching and learningstrategies; such a learning model may be applied althoughthe operator has no prior computer experience. By allowinglearners to be immersed in learning content through the useof virtual reality, drivers are allowed to apply informationfrom the test to product development (see Table 1).

3. Real-Time Detection of DriversUsing Augmented Reality

3.1. Obtaining Gesture Trajectory Information. Kinect sepa-rates human hands from the background environment as adivision strategy. OpenNI provides the API interface, thatis, a multilinguistic and cross platform framework that iscapable of communicationwith external hardware. Addition-ally, OpenNI includes an intermediate space for processinggesture recognition, tracking, and so forth. Currently, thereare four types of hardware supported by OpenNI: 3DSensor,RGBCamera, IRCamera, and Audio Device.

Figure 2 shows a picture that was acquired with a Kinectsensor. First, the skin color is detected; then, the hands aredeeply subdivided for separation into 2D and 3D. The handmodel is created in 3D, including an appropriately arrangedand assembled geometric primitive. Every hand gesture isrepresented by 27 vector points.With respect to estimation ofthe hand joints, 27 vector points are specified and tracked to

Mathematical Problems in Engineering 3

Figure 2: Schematic view for Kinect gesture tracking.

minimize different hypotheses between hands and real obser-vation of model parameters. To quantify such differences,the graphic rendering technology used generates similar skinand constructs depth mapping for given hands.Therefore, anappropriate objective function and a variance of PSO are bothdefined to obtain the optimal hand configuration. To outputthe result of such optimization, the hand joints are trackedwith time continuity.

3.2. Facial Feature Analysis. Real-time detection is also ap-plied to facial recognition using Kinect’s built-in SDK to cre-ate a facial recognition structure. The subsequent applicationincorporates this facial recognition structure into Unity tocreate an interactive function between the user and the virtualsystem (as shown in Figure 3).

3.3. Eye Recognition Analysis. Eye recognition generally in-cludes preprocessing, feature extraction, and sample learningand recognition.Many technologies andmethods can be usedfor eye recognition, including the following.

(i) Projection: here, eye location is detected according tothe projection distribution feature of a picture in acertain direction. Projection is a statistic method thatutilizes the eye’s gray-level information to detect theordinate and abscissa of the pupil through horizontaland vertical projections to obtain an accurate positionof human eyes. The most common projection func-tions are the integral, mean, and variance projectionfunctions.

(ii) Hough transformation: Hough transformation trans-forms pictures from a spatial domain to a parameterspace. Certain parameter forms satisfied by mostboundary points are used to illustrate a curve in apicture. The pupil is regarded as a standard circle.We may locate the eye’s pupil accurately using Houghtransformation through the standard equation of acircle. The robustness of Hough transformation isenhanced considerably because of significant geomet-ric analyticity: (𝑥 − 𝑎)2 + (𝑦 − 𝑏)2 = 𝑟2.

Figure 3: Schematic view of Kinect facial recognition.

(iii) AdaBoost classification: in the machine-learningfield, theAdaBoost algorithm is a high-efficiency, iter-ative computing algorithm that trains different weakclassifiers for the same training set, followed by gath-ering these weak classifiers to construct a strong clas-sifier. This algorithm is advantageous of its higherclassification accuracy and rapid human eye recog-nition. However, the effectiveness of the algorithmdepends on the selection of weak classifiers. It is veryimportant for the application of rapid human eye de-tection.

(iv) Sample matching: the location of the pupil may besearched for dynamically from left to right and fromup to down in a different picturewindowusing a circlesample according to pupil shape. Sample matchingsearches for a small picture of interest within a largerpicture. It identifies the location of the target withthe most similar location as the matching point bycalculating the similarity of the sample and matchingzone. A sample matching algorithm exists within thescope of the machine-learning field and is an effectiveeye recognition algorithm.

3.4. Gesture Trajectory Creation. The gesture trajectory rec-ognition used in this study is based on the hidden Markovmode (HMM).

HMM Model Definition. HMM is a very mature technologythat matches time-variant data. It is a random state machine.In a gesture recognition system, the observations of thevector sequence and coded symbolic sequence are jointlycalled the “observation sequence” (designated “𝑂”), whichis a random sequence. An HMM with 𝑁 states (designatedas 𝑆1, 𝑆2, . . . , 𝑆

𝑁) is represented by a triple parameter, 𝜆 =

{𝜋, 𝐴, 𝐵}, wherein

(i) 𝜋 = [𝜋1, 𝜋2, . . . , 𝜋

𝑁] is the initial distribution for

describing the probability distribution of observation

4 Mathematical Problems in Engineering

Image depth data

Hand 3D gridoverlap

Capturing human

information withNITE

Edge detection ofimage processing

for hand block

Sieving certainimage depth data

Sieving hand area

skeleton

Figure 4: Kinect hand recognition flow.

Table 2: Single factor variance analysis of individual aspect and entire aspect for different major use purposes with respect to the evaluationtest system of improvement for vehicle virtual reality.

Aspect Variation source SS df MS 𝐹

Goal motivationIntergroup 8.97 2 4.48 0.57Intragroup 769.21 97 7.93

Sum 778.18 99

Entertainment motivationIntergroup 216.84 2 108.42 10.65∗∗∗

Intragroup 987.46 97 10.18Sum 1204.33 99

Entire aspectIntergroup 317.36 2 158.68 2.67Intragroup 1193.1 97 59.37

Sum 1510.46 99∗∗∗𝑝 < 0.001.

sequence 𝑂 under the state at 𝑇𝑖and 𝜋

𝑖= 𝑃(𝑞

𝑖= 𝑆𝑖),

𝑖 = 1, 2, . . . , 𝑁, which satisfies Σ𝑁𝑖= 1𝜋𝑖= 1;

(ii) 𝐴 = {𝑎𝑖𝑗| 𝑖, 𝑗 = 1, 2, . . . , 𝑁} is a state transition

probability matrix; that is, 𝑎𝑖𝑗= 𝑃(𝑞

𝑖= 𝑆𝑗| 𝑞𝑖−1=

𝑆𝑖, 𝑞𝑖−2= 𝑆𝑘, . . .) = 𝑃(𝑞

𝑖= 𝑆𝑗| 𝑞𝑖−1= 𝑆𝑖), which

satisfies Σ𝑁𝑖= 1𝑎𝑖𝑗= 1;

(iii) 𝐵 is a state output probability matrix, which indicatesthe distribution of random variables or random vec-tors in the observation probability space of variousstates and𝐵 = {𝑏

𝑗(𝑘), 𝑗 = 1, 2, . . . , 𝑁}, 𝑘 = 1, 2, . . . ,𝑀,

which satisfies Σ𝑁𝑖= 1𝑏𝑗, 𝑘 = 1.

The dynamic gesture trajectory feature selects and ex-tracts dynamic gestures that are different from static gestures.For static gestures, the change is only on a spatial locationand morphology, whereas, for dynamic gestures, the changerelates to both time and space. The experiments show thathuman motion may be recognized only by motion infor-mation; therefore, gestures may be transformed into spatialmotion trajectories for recognition.

The basic features of a gesture trajectory are location,speed, and angle. Speed is not practical for our use becausethere can be large differences in speed for the same gestures.The coordinate locations of points on a gesture trajectorycurve may be significant features for gesture trajectory

recognition; however, differences are still present even forthe same gesture trajectories. Thus, there is a change incoordinate points. The change in the tangential angles of agesture trajectory at different times is used as a trajectoryfeature. With Kinect, the sensor may acquire palm locations

𝜑𝑡= tan−1 (

𝑦𝑡𝑐− 𝑦𝑡−1𝑐

𝑥𝑡𝑐− 𝑥𝑡−1𝑐

) . (1)

According to the obtained angle, a 12-direction chain code isused for discretization (see Figure 4).

We then use the Unity software to create a virtual realitysystem in conjunction with using various scenarios to buildvirtual reality streets and scenarios. Real vehicle operation issimulated through virtual reality, and environmental (capa-bility of simulating special weather, such as rain and snow)and human (e.g., a pedestrian crossing the road suddenly,appearance of an automobile or a motorcycle from a deadsector, or appearance of roadblock) variables are used ascriteria for building the system. Furthermore, it is builtaccording to potential sudden conditions in vehicle travelingin conjunction with human factor engineering design (seeTable 2).

After the system is built, Unity3D applies the virtualreality scenario to augmented reality in combination withdriver detection.The driver operationmodel is available from

Mathematical Problems in Engineering 5

Figure 5: Detection of the driver’s real-time posture.

Figure 6: Virtual reality scene.

the driving process of a scenario test, and the driver’s postureand gaze angle may also be used as test criteria.

Presently, people’s requirements of visual enjoyment areincreasing with the development of 3D stereo vision (asshown in Figures 5 and 6). For people to perform the testin different environments using virtual reality scenarios andhave a realistic feeling, environmental conditions simulatingreal scenarios may be created. Virtual reality simulation isnot only safe but also convenient. Furthermore, the accuracyof tests conducted with virtual reality is not affected byenvironmental factors that may be induced by tests in the realworld.

4. Conclusion

Streetscape is the main scenario of this study. 3D drawingsoftware is employed for modeling and mapping to drawthe entire street and for building models. After all buildingmodels are drawn, they are imported to Unity to create avirtual reality interface. For greater accuracy and multidi-mensional data collection, a realistic streetscape is developed,complete with a variety of sudden conditions involvingvehicles, pedestrians, roadblocks, and animals. Kinect is usedto perform augmented reality effects and improve the systemdesign for feedback regarding the operating behavior of thedriver. New 3D technology and program design are utilizedto construct lifelike road environments and traffic char-acteristics, including dynamic traffic flow and cross signalcontrol. Planned future work for additional application is toresearch the interactions between people, vehicles, and roads;for example, suitable vehicles, road systems, and situationswill be studied. Furthermore, both software and hardwarewill be updated to benefit from the increasing reality ofsimulated environments and for future extensibility of thedriving simulation system.

In the project, a real streetscape is simulated, followedby an evaluation of driver behavior and vehicular products.A variety of different visual effects are easily available toeffectively reduce error rates. Moreover, the reliability andreality of this project may be improved and the test securitymay be increased considerably. Different situation designs aresimulated and evaluated to increase development efficiencyand add more security verification test platforms using suchtechnology in conjunction with driving tests, vehicle fittings,environment factors, and so forth.

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper.

References

[1] G. Tsechpenakis, K. Rapantzikos,N. Tsapatsoulis, and S. Kollias,“A snake model for object tracking in natural sequences,” SignalProcessing: Image Communication, vol. 19, no. 3, pp. 219–238,2004.

[2] W. Admiraal, J. Huizenga, S. Akkerman, and G. T. Dam,“The concept of flow in collaborative game-based learning,”Computers inHuman Behavior, vol. 27, no. 3, pp. 1185–1194, 2011.

[3] L. M. Bergasa, P. F. Alcantarilla, and D. Schleicher, “Non-linearity analysis of depth and angular indexes for optimalstereo SLAM,” Sensors, vol. 10, no. 4, pp. 4159–4179, 2010.

[4] A. Carleton-Hug and J. W. Hug, “Challenges and opportunitiesfor evaluating environmental education programs,” Evaluationand Program Planning, vol. 33, no. 2, pp. 159–164, 2010.

[5] G. Klancar, M. Kristan, and R. Karba, “Wide-angle cameradistortions and non-uniform illumination in mobile robottracking,” Robotics and Autonomous Systems, vol. 46, no. 2, pp.125–133, 2004.

[6] P. KaewTrakulPong and R. Bowden, “A real time adaptive visualsurveillance system for tracking low-resolution colour targetsin dynamically changing scenes,” Image and Vision Computing,vol. 21, no. 10, pp. 913–929, 2003.

Submit your manuscripts athttp://www.hindawi.com

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

MathematicsJournal of


Mathematical Problems in Engineering

Hindawi Publishing Corporationhttp://www.hindawi.com

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of


Probability and StatisticsHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of


CombinatoricsHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

International Journal of


Operations ResearchAdvances in

Journal of


Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

International Journal of Mathematics and Mathematical Sciences


The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014


Algebra

Discrete Dynamics in Nature and Society



Decision SciencesAdvances in

Discrete MathematicsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com

Volume 2014 Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Stochastic AnalysisInternational Journal of

research article development of a real-time detection...

Documents