real-time markerless square-roi recognition based on contour-corner for breast augmentation
TRANSCRIPT
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
1/6
Real-Time Markerless Square-ROI Recognitionbased on Contour-Corner for Breast
Augmentation L. Rechard, and A. Bade
Abstract Augmented reality (AR) can be described as a technology consists of computer vision and image processing methods which is
used to enhanced user vision and perception. Whereas marker detection and recognition are the two most important steps which are vital in
the field of augmented reality (AR) in order to successfully placed computer generated object into the real world. This paper, aims to present
a real-time markerless identification and verification technique by combining contour-corner approach to detect and extract strong interest
point from the environment as the fundamental component in registering the virtual imagery with its real object without the needs to use
conventional printed marker. To enhance and to combine the current contour and corner detection approach, we proposed smoothing and
adaptive thresholding technique to the input stream and then use subpixel corner detection to obtain better and accurate interest point. Our
method used marker-less approach as to avoid the needs to prepare the target environment and to make our approach more flexible. The
proposed method starts with first getting an input from the real environment thru a camera as visual sensor. On receiving an input image, the
proposed technique will process the image, detects strong interest points from the ROI and recognize a marker by applying enhanced
contour-corner detection.
Index Terms Augmented reality, contour detection, corner detection, interest point
1 INTRODUCTION
ugmented reality (AR) can be described as a technologywhich is developed by using computer vision techniquesand image processing methods in order to enhanced user
vision and perception. Milgram et al. [1] defined AR as aresearch field in which users vision is enhanced by combiningreal world scene and computer generated objects into the
identical real environment space called mixed reality (see Fig.1) Whereas, Azuma (1997) [2] defines an AR as a system thathave the following characteristics:
Combines real object with virtual objects,
Interactive in real-time, and
Registered in three dimensions.
Fig. 1 Milgrams Reality-Virtuality Continuum [1].
Later, in 2002 Mann [3] come up with a two-dimensionalreality-virtuality-mediality continuum as an effort to addanother axis to Milgrams virtuality-reality continuum byadding mediated reality and mediated virtuality (see Fig. 2). In[3], the system can change reality in three ways:-
Add something (augmented reality)
Remove something (diminished reality) or
Alter the reality in some other way (modulated reality)
Fig. 2 Manns reality-virtuality-mediality continuum [3].
Various studies have been conducted to explore thfundamental concept and unique potential of AugmenteReality technology to be applied in mainstream applicatiosuch as in medicine, visualization, maintenance, path planninentertainment and education, industry, military and aircranavigation and yet most of it is marker-based application duto its ease of development and higher rate of accuracy an
success despite its lack of flexibility in unprepareenvironment. In the field of medicine, the use of computegenerated images such as CT-scan or MRI as a vision aid to thsurgeon in surgical planning has increased dramatically ovthe years. With the use of an AR system, we believe that thsurgeon's vision will be enhanced, more data can be obtaineensure proper surgical planning thus avoiding unnecessarcutting in real surgical operation.
Computer vision (CV) on the other hand is a field which strongly related to AR application development and succesAccording to Marr and Nishihara (1978) [4], vision is process that produces, from images of the external world, description that useful to the viewer and not cluttered witirrelevant information. In AR application, CV works hand hand with scene understanding and analysis. Sub-areas of C
A
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 15
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
2/6
can be seen in motion detection, face detection, object trackingand recognition etc. The process to correctly combine the realscene with virtual objects and accurately register the mixedscene depends greatly on CV potentials to extract and detectfeatures either naturally from the desired region of interest(ROI) or based on the fiducial marker that present to thecamera. Idris et al., (2009) [5], considered the use of CVtechniques as a starting point in detecting a fiducial marker ornatural marker in order to solve the registration and trackingissue in AR application.
Most AR systems operate based on the printed squaremarker (see Fig. 3) or customized pattern which is locatedphysically on top of the target area or workspace.
Fig. 3. Example of Printed Square Markers.
Therefore, in this paper, we proposed and describe areal-time markerless identification technique designed tocapture real scene, detect strong interest points from extractedcontour, verify a marker and overlay 2D object. Extracting anddetection of features from the real scene will be our concerns inrecognizing a marker.
Our proposed technique uses the combination of contourand corner approaches in order to find unique natural featuresfrom captured frame through a web cam in real time.Generally, feature points [6], edges [7], curves [8], planes andso on, are used for registration. Even though, extraction of suchnatural feature is more difficult than the artificial vision markerbased approach, the users movement range is not limited,ensure real scene augmentation and no preparation needed to
set up the marker. Therefore it is preferable to use thisapproach over the printed marker. The remainder of thispaper, will be discussing about the basic setup of AR system,marker and marker-less detection, contour-corner approachand results for the presented methods will also be given andillustrate.
2 RELATED WORKS
2.1 Simple AR system setup
The basic setup of an AR system consists of the mobilecomputing unit such as PDA or mobile phone andhead-mounted display (HMD) to which a miniaturized camera
is attached. There are two types of HMD generally used;namely optical see-through (see Fig. 4) and video see-through(see Fig. 5).
Fig. 4 Optical See-through Augmented Reality Display [9].
Fig. 5 Video See-through Augmented Reality Display [9].
The miniaturized camera therefore captures the reaenvironment from the users perspective. The AR systemprocesses these live images in real time extracting features toidentify markers. Markers are identified using point and edgedetectors. The features can be artificial printed markers, naturamarker or natural features.
2.2 Vision-based AR
Researchers in computer vision have proposed and developedseveral methods in tracking and detection that can be applied
in AR registration. This method can be sub-divided into threetypes based on the equipment used; sensor-based, visual-basedand hybrid-based. Since the camera was already part of the ARsystem, visual-based methods are used together withvision-based registration technique.
Approaches for visual-based tracking are marker-based (seeFig. 6) and markerless-based method. Used of marker-basedmethods have been discussed in [10], [11], [12], [13]. Whereas[14], [15], [16] have established a markerless-based method forunprepared environment. Most researchers opted for markerapproach, since it is more accurate and reducecomputation-resources [17]. However, for outdoor orunprepared environment, markerless approach is more
suitable despite suffered from computation time-consuming[18, 19].
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 16
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
3/6
Fig. 6 Marker-based ARToolkit Algorithm [20]
In order to be able to detect interesting features from theenvironment, [21], [22], [23] have proposed and developed
corner detectors techniques for visual tracking.
2.3 Corner detection
Corner or interest point detection is a processing stage incomputer vision. Literally, corner point is some kind of visualcorner, where two edges intersect. A good corner detector mustbe able to distinguish between true and false corners, reliablydetecting corner in noised image, accurately determining thelocations of corners and can be utilized in real-timeapplications (Shi and Tomasi, 1994).
Our method is based on Shi and Tomasi detector. Thisdetector is inspired by Harris detector (Harris and Stephens,1988) and its uses smallest eigenvalue. The difference lies in the
cornerness measure or selection criteria which made thisdetector better than the original. The cornerness for Harris wascalculated in (1) as given by Harris and Stephens (1988).
(1)
There are two ways to define corner:
Large response
The locations x with R(x) greater than certain threshold.Local Maximum
The location x where R (x) is greater than those of their
neighbors.
For Shi-Tomasi, it is calculated like this:
(2)
If R(x) is greater than a certain predefined value, it can bemarked as a good feature or a corner.
Classification of feature points using eigenvalues
Fig. 7 Shi-Tomasi detector [24]
Fig. 8. Harris detector [24]
As depicted in Fig. 7, if both eigenvalues are large, then thfeatures vary significantly in both directions and considered aa good feature (corner-like).
Traditional Corner detection flow chart
Fig. 9 shows a general flow chart of a corner detection process
Fig. 9 Corner detection flow chart [25]
2.4 Contour detection
The reason to perform contour detection in general, is significantly reduce the amount of data in an image bdetecting edges in a robust manner. Edge detection defines thboundaries between detected regions in an image. Thinformation needed for higher level image processing. Ahmaand Choi (1999) [26], and Ziou and Tabbone (1998) [27] havdiscussed different types of edge detection method. Our met
od is based on algorithm developed by Canny (1986) [28] sin
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 17
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
4/6
it produce smooth and continuous edges despite sensitive tonoise pixels and higher in computation-time [29].
As depicted in Fig. 10, there are several stages in this algo-rithm as stated in [28].
Fig. 10 Traditional Canny edge detection Algorithm [30]
3 APPROACH TO MARKER-LESS SQUARE-ROI
RECOGNITION
As demonstrated in Fig. 6, marker-based detection algorithm
needs to go through stage (d) and (e) respectively in order toimplant an object for the detected marker. These two steps arethe main inspiration for our proposed method. By combiningContour-Corner approach we believe that we can somehowreplace the needs to have a physical marker in the desiredenvironment. Instead of using a physical marker, we willdefine a Region of Interest (ROI) as a markerless marker, calledSquare-ROI. This Square-ROI need to be hand-drawn manuallyby the user (see Fig. 13). The reason we chose a square isbecause squares naturally produces 4 possible points and these4 points are needed to calculate the pose camera estimation forvisualization rendering purposes. Another reason is that, theorientation of the points can be estimated as intersections of
edge lines. To enhance and to combine the current contour andcorner detection approach, we proposed smoothing and adap-tive thresholding techniques to the input stream and then usesubpixel corner detection to obtain better and more accurate
interest point.
Fig. 11 Framework for Proposed system
As discussed in [31], a markerless recognition system needto fulfill some criteria as stated below:
Easy offline preparation
Automatic initializationAble to detect natural features reliablyReasonable computation timeAccurate augmentationWork in unconstraint environmentFlexible and adaptive to the application needs
3.1 Algorithm Architecture
As illustrated in Fig. 11, we can see that the process startedwith first getting an input from the real environment thru acamera as visual sensor. On receiving an input image, theproposed system finds and detects strong interest point fromthe ROI by applying enhanced contour-corner detection. Fromthe ROI, features such as number of corners and vertices can beextracted and later used to define a marker.
(a)
(b)
Fig. 12 Overview of the proposed method (a) Architecture (b) Marke
identification Illustration (i) draws detected contour (ii) draw detected corne(iii) Identify marker (iv) overlay an object [6]
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 18
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
5/6
4 RESULTS
4.1 Testing Setup
We present the results of our proposed method based on thetest conducted on a mannequin as an input (real environment)in our experiment.
A square is manually hand-drawn on top of the target area
as a region of interest (ROI) in order to place a non-printedmarker or we can say a markerless ROI (see Fig. 13 (a)). Therationale to manually hand drawn a square contour as ROI isto avoid the needs to prepare a printed marker and at the sametime to make our proposed system more flexible withunprepared environment. Thus, we can ensure that onlydesired area of the image is searched for features. As visualizedin Fig. 13 (a), our method will first capture a frame from avideo feed, binaries captured frame, find contour in squareshape, detect interest point and finally identify a marker.
We formulate that number of vertices detected on a contour(square) must be equal with the number of corners detected in
order to identify a marker and upon successfully identified ayellow circle will be overlaid on top of the target area (see Fig.13 (d)). Yellow circle used as a substitute for 3D breast object inthis test.
Fig. 13 Results of the proposed system: (a) Load camera and capture the
environment (b) Convert captured frame into grey-scale (c) Displaydetected corners and square (d) If marker detected (Overlay 2D Yellow
Circle) [6]
4.2 Execution
From our test, we found that there are three (3) factors couldcontribute or influence our proposed method success rate.
Square-ROI size Square-ROI line thickness and Square-ROI distance from camera (viewpoint)
In Table 1, it shows that, the size of Square-ROI used (witeither single or double smoothing) influence thDistance-to-detect features.
TABLE 1
Square-ROI size and Distance-to-detect
Square-ROI size Distance-to-detect Smoothing
17 x 16 (cm) Estimated at 77 (cm) Single17 x 16 (cm) Estimated at 40 (cm) Double
6 x 5 (cm) Estimated at 15 (cm) Single6 x 5 (cm) Estimated at 10 (cm) Double
5 CONCLUSIONBased on our experiment in section 4, the proposed techniqumanaged to do the followings:
Capture real scene through a camera.
Convert captured scene into a grey-scale image.
Detect four (4) vertices and four (4) corners.
Identify and verify a marker based on the extractefeatures
Overlay 2D object. (Yellow circle)
At the moment, proposed technique have the tendency detect unwanted contour and corner as shown in Figure 13 (and required bigger memory (RAM) size estimated at 16GB foa duration of 7 to 14.1 seconds real-time execution.
In the future, we would like to extend our technique properly visualize the coexistence of a real and synthetic 3
breast cancer model that sharing the same real environmenwith the aid of touchless hand and finger interaction to selethe Region of Interest(ROI) in real-time.
ACKNOWLEDGMENT
The authors wish to thank the GRAVSLAB research group ftheir support and advice. This research is supported by a gran(FRGS0295-SG-1/2011) from the Ministry of Higher Educatio(MOHE), Malaysia.
This paper is an extension of work originally reported in "The2013 2nd International Conference on Medical Information andBioengineering (ICMIB2013).REFERENCES
[1] P. Milgram, H. Takemura, A. Utsumi and F. Kishino, Augment
Reality: A Class of Displays on the Reality-Virtuality Continuum
Proceedings of SPIE, Vol. 2351, Telemanipulator and Telepresen
Technologies, Hari Das; Ed. 1994. Pp. 282292.
[2] R. Azuma, Y. Baillot, R. Behringer, S. Feiner , S. Julier, an
B. MacIntyre, Recent advances in augmented reality, IEE
Computer Graphics and Applications ;21(6):3447, 2001.
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 19
-
7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation
6/6
[3] S. Mann, Mediated Reality with implementations for everyday life,
Presence 2002, Teleoperators and Virtual Environments.
[4] D. Marr and H. K. Nishihara, Representation and recognition of
the spatial organization of three-dimensional shapes," Proc. R. Soc. of
Lond. B, 200, 269-294, 1978.
[5] M.Y.I. Idris, H. Arof, E.M. Tamil, N.M. Noor and Z.Razak, Review
of Feature Detection Techniques for Simulataneous Localization
and Mapping and System on Chip Approach, Inform. Technol.
Journal, 8(3), pp. 250-262, 2009.
[6] Y. Genc, S. Riedel, F. Souvannavong, C .Akinlar and N. Navab,
Markerless tracking for Augmented Reality: A learning-based
approach, Proc. of the International Symposium on Mixed and
Augmented Reality (ISMAR 02), pp. 295-304, August 2002.
[7] L. Vacchetti, V. Lepetit and P. Fua, Combining edge and texture
information for real-time accurate 3d camera tracking, In
International Symposium on Mixed and Augmented Reality
(ISMAR 04), pp. 48-57, 2004.
[8] A. I. Comport, E. Marchand, and F. Chaumette, A Real-Time Tracker
for Markerless Augmented Reality, In International Symposium on
Mixed and Augmented Reality (ISMAR 03), Tokyo, Japan,
September 2003.[9] J. R. Vallino, Interactive Augmented Reality, PhD Thesis,
University of Rochester, Rochester, NY. November 1998.
[10]J. Rekimoto, Matrix: A realtime object identification and registration
method for augmented reality, In 3rd Asia Pacific Computer Human
Interaction (APCHI98), pages 6368, 1998.
[11]H. Kato and M. Billinghurst, Marker tracking and HMD calibration
for a video-based augmented reality conferencerencing system, In
Proceedings of the 2nd IEEE and ACM intl. workshop on augmented
reality (IWAR99) (p. 85), Washington, DC, USA, 1999.
[12]M. Fiala, ARTag, a fiducial marker system using digital tech niques,
In Proceedings of the IEEE conference on computer vision and
pattern recognition (CVPR05), vol. 2, pp. 590596, Washington, DC,
USA. 2005. doi:10.1109/CVPR.2005.74[13]S. Sanni, Theory and applications of marker-based augmented
reality, Espoo 2012. VTT Science 3. 198 p. + app. 43 p.
[14]I. Skrypnyk and D. G. Lowe, Scene modelling, recognition and
tracking with invariant image features, In Proceedings of the 3rd
IEEE and ACM intl. symposium on mixed and augmented reality
(ISMAR04) (pp. 110119). 2004. doi:10.1109/ISMAR.
[15]G. Klein and D. Murray, Parallel tracking and mapping for small AR
workspaces, In Proceedings of the 6th IEEE and ACM Intl.
Symposium on Mixed and Augmented reality (ISMAR07), Nara,
Japan. 2007.
[16]T. Lee and T. Hllerer, Hybrid feature tracking and user interaction
for markerless augmented reality, In Proceedings of the IEEE virtual
reality conference (VR08) (pp. 145152). 2008. doi:10.1109/VR.2008.4480766
[17]F. Zhou, H. Duh and M. Billinghurst , Trends in augmented reality
tracking, interaction and display: A review of ten years of ismar , In
7th IEEE and ACM International Symposium on Mixed and
Augmented Reality (ISMAR08). 2008.
[18]D. Beier, R. Billert, B. Bruderlin, D. Stichling and B. Kleinjohann,
Marker-less vision based mobile augmented reality, Proc. tracking
of Japan, Oct 2003.
[19]G. Simon and M. Berger, Reconstructing while registering: a novel
approach for markerless augmented reality, Proc. of ISMAR,
Darmstadt, Germany. 2002.
[20]ARToolkit Library. Available
http://www.hitl.washington.edu/artoolkit/documentation/vision.h
tm. Last accessed 10 February 2013 at 10.46 pm.
[21]H. Moravec, Towards automatic visual obstacle avoidance, In
Proceedings of the International Joint Conference on Artificia
Intelligence, pp. 584. 1977.
[22]C. Harris and M. Stephens, A combined corner and edge detector,
Alvey Vision Conference, pp. 147151. 1988.[23]C. Tomasi and T. Kanade, Shape and motion from image streams : A
factorization method-part 3. Detection and Tracking of Poin
Features, Technical Report CMU-CS-91-132. 1994.
[24]D. Frolova and D. Simakov, Matching with Invariant Features,
Advanced Topics in Computer Vision, Spring 2004, Weizmann
Institute of Science. March 2004. Lecture slide.
[25]D. Parks and P. Gravel, Corner detection, Available
http://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htm.
Last accessed 5th March at 2.17 am.
[26]M. B. Ahmad and T. S. Choi, Local Threshold and Boolean Function
Based Edge Detection, IEEE Transactions on Consumer Electronics
vol. 45, no 3, August 1999.
[27]D. Ziou and S. Tabbone, Edge detection techniques: An overview,
International Journal of Pattern Recognition and Image Analysis
8(4):537559. 1998.
[28]J. Canny, A computational approach to edge detection, Pattern
Analysis and Machine Intelligence, IEEE Transactions on
PAMI-8(6):679698, Nov 1986.
[29]S.K. Mahendran, A Comparative Study on Edge Detection
Algorithms for Computer Aided Fracture Detection Systems ,
International Journal of Engineeirng and Innovative Technology
(IJEIT), vol. 2, no. 5, November 2012.
[30]L. Rechard, A. Bade, S. Sulaiman, & S. H. Tanalol, Using Compute
Vision Techniques to Recognize Hand-Drawn Square-ROI as a
Marker in Augmented Reality Application, In Proceedings of the
10th Annual Seminar on Science and Technology 2012 (SNT 2012)
p.212, 2012.[31]Y. Chunrong, A tracking by detection approach for robus
markerless tracking, Available
http://campar.in.tum.de/twiki/pub/Chair/IsmarIndustrialAugmen
tedReality2006/yuan.pdf
Rechard Lee holds a first degree in Computer Science from UniversityPutra Malaysia completed in 2000. He is currently pursuing his MSc degreeby research in Mathematics with Computer Graphics at University MalaysiaSabah. Current Research Group is GRAVSLAB (www.gravslab.com)
Dr. Abdullah Bade holds MSc degree by research in Computer Sciencefrom Universiti Teknologi Malaysia in 2003. He obtained his Doctoratedegree in Industrial Computing (Computer Graphics and Visualization) fromUniversiti Kebangsaan Malaysia in 2008. He has been actively involved in
research and manages to secure several research grants from Ministry oHigher Education and Ministry of Science and Technology Malaysia. Hisresearch interest lies particularly in developing optimized algorithm focollision detection between objects, Deformable Body Simulation, SeriousGames Simulation,Cloth Simulation and Crowd Simulation. Currently, he ispart of the School of Science and Technology, UMS and appointed as aSenior Lecturer and Head of Programmes for Mathematics withComputer Graphics. He spends most of his leisure time on listening sofmusic, surfing internet, reading and travel. Current Research Group isGRAVSLAB (www.gravslab.com)
JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 20
http://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htmhttp://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htm