JOHN Y. C. WANGU. S. Army Engineer Topographic Laboratories
Fort Belvoir, VA 22060
Feature Extraction byInteractive Image Processi ng
The computer aids the operator in detecting and delineatingcultural features on digitized imagery.
INTRODUCTION
A N IMPORTANT SUBJECT before scientists, engineers, and cartographers at
the present time is the rapid, accurate, andeconomical processing of imagery that depicts the surface of the earth or other planetsremotely sensed from space vehicles. Inmapping we are interested in planimetric
the original picture into simpler form, e.g., aline drawing, by some sequence of steps.Each step should preserve the importantproperties of the original, while reducingthe computational requirements imposed onsubsequent steps. But the reduced picturegenerally contains less information than the
ABSTRACT: In aerial photography, images of roads and other manmade structures are small compared to those of natural features.Their edges are usually indistinct from the background and often goundetected by automatic systems. A major problem encountered byautomatic detection systems is the delineation ofboundaries aroundareas containing features of interest. The computer is often unableto accomplish the detection process alone. Therefore, it is reasonableto combine the superior pattern recognition abilities of a humanbeing with the computational power of a digital computer to forman interactive system. Such an interactive feature extraction systemis discussed. The objective is to generate the photo coordinates forfeature boundaries from aerial photographs. The hardware consistsof a CDC-6400 sequential computer, a Goodyear STARAN associative array processor, a DEC PDP 11/50 minicomputer, and an imagedisplay, as well as digitizing and recording devices. The featureextraction program for the interactive system consists of a series ofroutines which were designed, written, and installed in the systemby the author. The routines were written in FORTRAN and/or theSTARAN assembly language according to the intended applicationin order to use the computers efficiently.
features such as roads, built-up areas, drainages, and natural boundaries; however, tracing planimetric features from aerial photographs rapidly, accurately, and economicallyis not a simple task. Presently, automatedcartographic instruments can produce a variety of maps, but cannot compile line mapscompletely automatically. Cartographic details still must be extracted manually byhuman operators. The extraction of featuresis the bottleneck in the total automation ofthe mapping processes.
Image processing methods may convert
original, and there is no guarantee that theinformation lost is irrelevant. A number ofmethods to determine planimetric featuresautomatically from digital data may serve forsome types of imagery or for some specialpurpose3.4, however, there is no single standard approach to the extraction of featuresfrom remotely sensed data. In practice, it isnot always wise to attempt to map objectsinto clear-cut, two-value overlays. Very oftenobjects are defined only in a "fuzzy" senseand no definite or distinct edge can bedrawn by automatic methods.
PHOTOGRAMMETRIC ENGINEERING AND REMOTE SENSING,
Vol. 43, No. 12, December 1977, pp. 1495-1501.
1495
1496 PHOTOGRAM METRIC ENGINEERING & REMOTE SENSING, 1977
In aerial photography, roads and buildings are very small compared to fields andforests, and their edges usually are not distinct from the background. In small-scalephotography, roads are usually imaged as awhite line with a small surface area. Thesemay often go undetected by automatic systems while features of less interest to mapmakers, such as fields, are easily detected.Even with a good edge extraction method,there remains the problem of decidingwhich edges should be extracted to give themaximum amount of information in the finalmap. Feature extraction from digital imageryis, in general, a complicated problem, so thatthe computer is often unable to accomplishthe task alone. Therefore, it appears reasonable to combine the superior pattern recognition abilities of the human being with thecomputational power of the digital computerto form an interactive system. In this paperwe are concerned with the extraction offeatures by the interactive digital imagesystem in order to generate photo coordinates automatically. If we have the photocoordinates of features from stereo photography, we can use analytical photogrammetricmethods to generate ground coordinates andto compile line maps.
INTERACTIVE DIGITAL IMAGE
PROCESSING SYSTEM
An interactive digital system offers flexibility. Once the basic hardware and softwaresystems are developed, only minimal software effort is required to utilize the systemfor new applications. The interactive digitalimage processing system which was utilizedto conduct this research was developed atthe U.S. Army Engineer Topographic Laboratories, Fort Belvoir, Virginia. The hardware consists of a CDC-6400 sequentialcomputer, a Goodyear STARAN associativearray processor, a DEC PDP 1l/50 minicomputer, standard peripherals, and associated image display, digitizing, and recording devices. The CDC 6400, STARAN, andPDPll computers have their own systemsoftware package to handle all operations fortheir system. There are interface softwarepackages to allow all computers in the system to communicate with each other.
Images to be processed on the interactivedigital image processing system are input tothe system from magnetic tape or digitizedtransparencies made on the DICOMED056. Once the images are in the system,they are stored on CDC's 844 disks. Theinteractive capability is achieved by usersworking through the Tektronix 4014-1 alpha-
numeric console and the COMTAL imagedisplay unit. The computations are done inthe CDC-6400 computer and STARAN parallel processor that allows the programmerto take advantage of parallelism in the arithmetic structure of a problem. The finalresults can be put on hard copies, printed ontransparencies on-line by the DICOMEDD47, or off-line by copying to magnetic tapefor recording on other hard copy devices.
BASIC ALGORITHMS
A typical picture usually consists primarily of features having homogeneous densities and certain properties, such as size,shape, direction, orientation, and adjacencywhich are most impOitant for map makers. Ifwe can accurately determine the boundaries(edges) of the objects, we may be able tomeasure these essential properties. We assume that the boundary around an object is aclosed loop except as it reaches the limits ofthe data. The following algorithms to extractthe features by using the interactive digitalimage systems have been developed.
EDGE DETECTION
Initially, the user has to provide a startingpoint and a threshold value for detectingedges after selecting a feature for manipulation. The basic approach that we have designed consists of two steps. The first isfinding an initial point. The computer compares the gray level of one pixel at a time ofa small array, whose upper-left corner is thegiven starting point, against the input threshold value. If the threshold gray level lies between the gray levels of two adjacent pixels,an edge exists between the two pixels. If thetwo adjacent points are on the same line, theorientation of the edge is vertical across thelength of the pixel. If the two adjacent pointsare on the different lines, the orientation ofthe edge is horizontal.
The second step is detecting a new edgepoint by comparing the gray levels of threenearest neighbors of the initial point againstthe threshold value. It is done according tothe orientation (horizontal and vertical) ofthe edge. Suppose (see Figure 1) a horizontal edge point is at location @ betweenelements (i, j) and (i +1, j) and assume thegray level of element (i, j) ~ T (the thresholdvalue) and the gray level of element (i +1, j)~ T. If the edge has an extension, there arethree cases; the new edge point may be(a) between elements (i, j) and (i, j +1),(b) between elements (i, j +1) and (i +1,j +1), or (c) between elements (i+ 1, j +1)and (i + 1, j). Only two neighbor elements
FEATURE EXTRACTION BY INTERACTIVE IMAGE PROCESSING 1497
op j+1
~I• p
+ + + + +
+ a .1 +--~ II b
;+1 c .
R.
FIG.!. Detecting a new edge point from a knownhorizontal edge.
need be checked to determine the locationof the edge. If the gray level of element (i,j +1) ",:; T, it is case a. If it is not case a, andthe gray level element (i +1, j +1) ",:; T, it iscase b; if the gray level of element (i +1,j +1) > T, it must be case c.
If the previous edge is vertical and the lastedge point is at location ® (see Figure 2)and assuming the gray level of element (i, j);;" T and the gray level of element (i, j +1)",:; T. There are three cases. The edge may be(a) between the elements (i, j) and (i +1, j),if the gray level of element (i +1, j) ",:; T; (b)between the element (i + 1, j) and (i + 1,j +1), if it is not case a and the gray level ofelement (i +l,j +1) ",:; T; and (c) between theelement (i, j+l) and (HI, j+l), if it isneither case a nor case b.
LINE DETECTION
A line is a long and narrow region whichhas a certain width, homogeneous gray levels in the region, and a gray level distinctly
op
;+1
R.FIG. 2. Detecting a new edge point from a knownvertical edge.
different from its background. It is difficultto detect a line by the edge detecting method· but if we can determine the width andth~ central points of the line, we may beable to measure the essential properties.Initially the user must provide a startingpoint, the critical value (Ie) of the gray leveldifference between adjacent elements in theline region, after he has selected a linefeature for manipulation. There are twosteps for detecting. The first step is searching all parameters which are required in thesecond steps, such as the width, direction,orientation, and the gray level of the line.The second step is locating the centralpoints of the line by comparing the difference between the gray level of elements andthose of the background against the criticalvalue (IG) and also comparing the differenceof gray level between adjacent elementsalong the line against the critical value (Ie)according to the orientation of parameter ofthe line. If the difference of the average graylevel of the cross section of the line and theaverage gray level of background is greaterthan the critical value (IG) and the difference of the average gray level of two adjacent cross sections of the line is smaller thanthe critical value Ie, then the new points arethe line points and the central point of thecross section is recorded.
PROCEDURES
A stereo pair of aerial photographs #98and #99 over the Arizona Test Area wereused. The scale of photography is about1:47,100. The Model1050A Microdensitometer system was used to measure and scanthe images for this research. A point in theupper-left corner of the photo was chosen asthe origin of the photo coordinates system.For both photos, all fiducial marks, thereseau intersections of interest, and theupper-left corner of the mapping area whichwould be digitized were measured with thescanner in the manual mode, and the photocoordinates were recorded. An aperaturesize of 34.5 micrometers and the 5x magnification for the eyepiece and the objectivelens were chosen. Then scan line and pixelspacings were selected to be 30 micrometerseach. A square array (2048 x 2048 points) oneach photo was digitized, and the data wererecorded on magnetic tapes. After inputtinga portion of the digital data from the photography into the system and displaying it onthe screen, the user can examine the sceneand decide which feature will be extractedfirst. Then the subset of data which con-
1498 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1977
(b) If it is case b, the location of the newedge point is between element (i + 1,j) and (i+ 1, j + 1); then the localcoordinates of the point are
(c) If it is case c, the location of theedge point is between the elements(i+ 1, j + 1) and (i, j + 1); then thelocal coordinates of the point are
(c) If it is case c, the location of the newpoint is between element (H 1, j)and (H 1, j + 1); then the local coordinates of the point are
p = op + j + 0.5 * IFG
op and ol are the coordinates of theupper-left corner point of the datamatrix. IFG is the indicator of edgedirection. If IFG=I, the edge isfrom left to right. If IFG= -1, theedge is from right to left. The unit ofthe p and l is the pixel spacing of thedigitized data.
(5)
(4)
(3)
l = ol + i + 1.
l=op+i+1.
p = op + j + 0.5
l = ol + i + 0.5 * IFG.
and
and
and
For detecting an edge point from a knownvertical edge and a threshold value, the prinCipal detecting method and procedure arethe same as the known horizontal edge. Thedifference is the orientation of the knownedge; therefore, the comparison is done~ounterclockwiseand the ways for generatmg the local coordinates are different.
The local coordinates of edge points arecalculated as follows (see Figure 2):
(a) If it is case a, the location of the newedge point is between element (i, j)and (i + 1, j); then the local coordinates of the point are
p=op+j
tained the feature will be moved into thecomputer and displayed on the screen at alarger scale.
EDGE DETECTION
If an edge is to be detected, the usershould input a starting point and a thresholdvalue. The starting point can be entered intwo ways: one is the coordinates of thepoint, the other is the position of the cursoron the screen. There are three ways toobtain the threshold value: (1) Users willprovide the threshold value which must beinput to the system; (2) the program willprovide the threshold value by using themean gray level of the data; or (3) by plottingthe histogram of the gray levels of the dataand using the bottom value of the valley ofthe histogram.
The first step is to search for an initialedge point in a small array in which theupperleft corner is the given starting point.The computer compares the gray level ofone pixel at a time against the thresholdvalue. If the threshold gray level lies between the gray levels of two adjacent pixels,an edge exists between the two pixels. Afterfinding an edge point, the program will stopthe search and record the local coordinatesof the initial edge point for the edge and theorientation of the edge. The local coordinates of the initial edge point (refer again toFigures 1 and 2) are p = op + j +0.5 andl = ol + i for the vertical edge and p = op + jand l = ol + i +0.5 for the horizontal edge,respectively.
The second step is to locate the edgepoints according to the orientation of theedge. If there is a horizontal edge, the localphoto coordinates of the edge points arecalculated as follows (see Figure 1):
(a) If it is case a, the location of the newedge point is between elements (i,j) and (i, j + 1); then the local coordinates of the point are
p = op + j + 0.5 * IFG
and (1)
l = ol + i.
(b) If it is case b, the location of the newedge point is between element (i,j+I) and (HI, j+I); then the localcoordinates of the point are
p=op+j+I p=op+j+I
and (2) and (6)
l = ol + i + 0.5. l = ol + i + 0.5 * IFG.
FEATURE EXTRACTION BY INTERACTIVE IMAGE PROCESSING 1499
IW is the width of the line and g(i, k) is theleft nearest background neighbor of the linesection.
Equations 8 and 9 must be satisfied simultaneously. The cross section at i is a partof the line and the central point is at g(i, k+ (IW/2)). If some cross section fails theEquation 8 test and passes the Equation 9test, the program will not register the crosssection as a part of the line and will processthe next cross section until some sectionfails both tests, then the program terminatesthe line.
The program will generate the local coordinates for the central points of a line byusing Equation 10.
where ap and 01 are the local coordinates ofthe upper-left corner of data matrix, and IWis the width of line.
The principles of and procedures for detecting other orienting lines are the same asthose for the vertical line. The only difference is in the equations for testing andcalculating the local coordinates; therefore,that the details are omitted.
All edge or line points are displayed onthe screen by an overlay. If no additionalnew points for the edge or line can be located, or the edge or line has reached thelimit of the data, the process is paused.Operator assistance is needed to give a newstarting point or perform other operations. Ifa boundary of an edge or line is terminated,the local coordinates can be output. However, because the edges or lines are usuallysmooth curves, the two-dimensional raw
(10)
IW
I = 01 + i
p = ap + k + IW/2
op;'-_-..£::.::::::::::::~---+p
01 0 x x:x x x 0
0 x x x x x 0
g(; ,k(~x x@x x 0
x x x x x 0
0 x x x x x 0
0 x x x x x 0
0 x x x X x 0
0 x x x x x 0
0 x x x x x 0
0 x X x x x 0
0 x x x x x 0
0 x x x x x 0
0 x x x x x 0
0 x x x x x 0
1
FIG. 3. The vertical line.
and
ap and 01 are the coordinates of theupper-left corner point of the datamatrix. IFG is the indicator of edgedirection; if IFG = 1, the edge isfrom top to bottom; if IFG = -1, theedge is from bottom to top. The unitof the p and I is the pixel spacing ofthe digitized data.
IW (9)-I g(i+l,j+k)/ ,;;;IC
j ~ 1
LI E DETECTION
If a line is to be detected, the user shouldprovide a starting point and two criticalvalues (IG and Ie). The program will firstsearch the area around the starting point togenerate the property parameters such aswidth, direction, orientation, and the initialcentral point of the line by counting theelements whose difference of gray levels issmaller than the critical value IC. If thehorizontal length of the region is longer thanthe vertical length, the line is orientatedhorizontally and the vertical length is thewidth of the line; the middle point of thecross section is the central point of the line.If the horizontal length is equal to thevertical length, it means the line is a 45degree tilted line. The parameters for thetilted line are obtained in the same manneras for the horizontal or vertical line. Afterthe program has all parameters, it detectsthe line according to its orientation. If it is avertical line, the average gray level of thehorizontal elements of the line (the crosssection of the line) is compared to the average of two background neighbors. If thedifference of the average gray level of theline and average gray level of the background is greater than the critical value IGand the difference of the average gray levelof the line and average gray level of the background is greater than the critical value IGand the difference of the average gray level oftwo adjacent cross section of line is smallerthan the critical value IC, then the newpoint is the line point (see Figure 3). Themathematical expressions are as follows:
1 IW
1- I g(i,j+k)IW j = 1
and
- ~ [g(i, k) + g(i, k+IW)J I ~ IG (8)
1 IWIrw I g(i,j+k)
j = 1
1500 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1977
data (local coordinates) might be smoothedas a straight line or a second-order curveaccording to their shape by a least squareadjustment program. After adjustment, theresults are displayed on the screen; then theuser may decide how many points will beused and/or stored as the local coordinatesfor representing the feature of the photograph.
The local coordinates of correspondingfeature points can be obtained by using adigital image match technique. An independent array (11 x 11) whose central point isthe known feature point will move aroundon the corresponding image to find a match.The correlation values are computed according to the following equation for matching:
opPo£
6P~
6£
1- + + +£0
+ 0-, +0d
+ + +
£
FIG. 4. Two dimension correlation function andthe match point.
PHOTO COORDINATES
The photo coordinates of features aregenerated:
GGAT (m, n) (11)
=II(AD *TD )/(VAR[A] *VAR[T]lij
where A is the dependent array, T is theindependent array,
and
x = ULX + p * s - ox
(13)
ij
and
AD = [A(i, j) - AM],TD = [T(i - m,j - n) - TM],
Var [A] II (AD)2,
y = -(VLY + 1* s - oy).
Where ULX and VLY are the plate coordinates of the upper-left corner ofthe digitizedarea; p and 1 are the local coordinates offeature; s is the scale factor (the pixel spacing); and ox and oy are the plate coordinatesof the principal point. The photo coordinatesof the features accompanying descriptivetitle or heading on each file and a series ofcodes for each line segment are a part of theoutput. The codes will indicate the type andkind of features. One code will indicatewhether the feature is an edge or a line. If itis a line, the width of the line should berecorded, since the line detecting routineonly gives the coordinates of the centralline. If it is a closed boundary, a codeindicating the texture within the region isneeded. Another code will indicate whetherthe data represent a straight line or a curve.If it is a curve, the mathematical model isused for the least square adjustment routine.From the codes, the map compiler can determine precisely the characteristics of thefeature without referring to the photograph.
(12)
I' = 1 + !i.l + ff.1.
p' = p + !i.p + ff.p
and
Var [T]
Var [A] and Var [T] are variances of A and T,and AM and TM are the mean of A and T,respectively.
The routine then finds the maximum valueamong the correlation values and its eightnearest neighbors, and it fits a second-ordersurface through the nine points by a leastsquare adjustment routine. The adjustedpeak of the surface is the matched point (seeFigure 4).
The coordinates of the matching point are:
where p and 1 are estimated coordinates, !i.pand !i.1 are shifts between the estimated pand 1 coordinates of the largest computedcorrelation location, and ff.P and ff.l are thecorrections of coordinates from the leastsquares adjustment. p' and I' are the finallocal coordinates for the corresponding feature points.
CONCLUSION
The tracing of features manually in greatdetail is time-consuming and requires skilledoperators. Since these tasks are routine,repetitive operations that require great precision, it would be desirable to automate theentire process. However, many of these
FEATURE EXTRACTION BY I TERACTIVE IMAGE PROCESSING 1501
tasks require perceptual abilities and areexpected to continue to be beyond the capabilities of a fully automatic system for thenear future.
An alternative approach is to combineabilities of man and machine-an interactivesystem. This approach has several advantages.
• Decisions can be made and adapted to theproblem as frequently as the operator likes.When errors are committed, he may interrupt the process to make corrections and/orto improve the results until he is satisfied.
• Techniques can be selected depending onthe context. If a routine procedure does notwork in a particular situation, the operatormay try various techniques to process thedata. This flexibility gives researchers avery useful tool for experimenting withvarious techniques and procedures for evaluating data for exploratory development.
• Only the data of immediate interest to theuser will be handled. As a result, less spacewill be required and computer time willbe reduced. The computations which areinherently parallel in nature were performed on the STARAN parallel processorto speed up the processing.
This paper demonstrates the capability forextraction of cartographic detail from digitalaerial photographic data. It appears to offera substantial improvement in speed over themanual method. Also, its interactive capability between operator and computer offersa great potential for applications in otherfields.
REFERENCES
1. Crombie, Michael A., Semiautomatic PassPoint Determination Using Digital Techniques, ETL-005I, U.S. Army Engineer Topographic Lab., Ft. Belvoir, VA 22060, December 1975.
2. Gambino, L. A. and M. A. Crombie, "DigitalMapping and Digital Image Processing,"Photogrammetric Engineering, November1974.
3. Ketting, R. L. and Lundgrebe, AutomaticBoundary and Sample Classification of Remotely Sensed Multispectra Data, LARSInformation Note 041773, LARS PurdueUniversity, April 1973.
4. VandeBrug, G. J., "Line Detection in SatelliteImage," Machine Processing of RemotelySensed Data Symposium, Proceedings, LARSPurdue University, June 1975.
Call for papers
Seventh AnnualRemote Sensing of Earth Resources Conference
Tullahoma, TennesseeMarch 27-29, 1978
Sponsored by the University of Tennessee Space Institute, the technical program willfeature papers reflecting recent developments in
• Remote Sensing Techniques• Land Use Planning• Exploration and Management of Natural Resources• Agricultural Surveys and Planning• Forest Inventory• Air and Water Pollution Measurement and Analysis• Wetlands and Soil Moisture Mapping• Instrumentation and Sensor Equipment• Photo Interpretation and Photogrammetric Applications
Those wishing to present papers please submit the title of the proposed paper; the author'sname, address, and position; and a 140-word abstract by January 10, 1978 to
Dr. F. ShahrokhiConference DirectorThe University of Tennessee Space InstituteTullahoma, Tennessee 37388