FRONTISPIECE. Lunar surface as taken by Surveyor 1. The view on the left is BeforeEnhancement, and 011 the right is . Iftel' Enhancement.

ED\L\RD EFRON*

IBM CorporationLos A ngeles, Calif. 90041

Image Processing byDigital SystemsThe resultant pictures will be displayed with clarityand detail that could not normally be seen.

I :\TRODUCTIO:\

T HE PURPOSE OF this paper is to describe atechnique by \\'hich tele"ision pictures

can be stored, processed and pictorially dis­played by use of a digital computer system.The system described, which will be referredto as the I mage Processing System (IPS), willaid the photointerpreter (user) in the diag­nosis of an image in an on-line, multipro­gramming, man-computer cOI1\'ersational andti me-shari ng mode.

The IPS will automatically read pictorial in­formation that may be present on an analogtape (similar to a video tape), or in normalphotographic form. It \\'ill then digitize the

• Presented at the Annual Convention of the:\merican Society of Photogrammetry, \\"ashing­ton, D. c., :'Iarch 1968. The work was performedfor the Jet Propulsion Laboratory, CaliforniaInstitute of Technology, sponsored by the NationalAeronautics and Space Administration under Con­tract NAS7-100.

image, extract significant information, mathe­matically manipulate it, compare it withpreviously stored data and available knowl­edge, and pictorially display on a cathode raytube (CRT) the information that the photo­interpreter desires to view. The photointer­preter will be able to communicate with thesystem "ia a light pen, a keyboard, and atypewriter to ask questions and make re­quests. The predously stored computer pro­grams may remO\'e digitally the noise createdduring the teleyision and transmission pro­cesses, and can apply yarious digital filters toreturn the recei"ed signals to their originalstrength (enhancement of the pictures). Theprograms can also correct for other televisionimperfections \\'hich cause geometric distor­tion and intensity errors. The resultant pic­tures will be displayed with clarity and detailthat could not normally be seen. At the re­quest of the photointerpreter, in an on-line,man-com pu tel' conyersational mode, addi­tional picture processing can be performed

1058

DIAGE rHOCESSl:\G BY DIGIT,\L SYSTDrS 1059

such as enlargement of sections, as well ascontour map generation and pattern recogni­tion to loca te and pcrform I1lcasu remcn ts onvarious picture features.

The multiprogramming modc and time­sharing capability consist of a backgroundand multi-user mode, respecti\·cly. The back­ground mode \"ill permit the normal produc­tion job to be processed \\'hile the user is atthe console thinking about his next operationand not using the computer. \\"hen the user is

system basically consists of two parts, thehardware and the software. The followingsubsections will handlc cach scparatcly.

HARDWAllE

The IPS cousists of an IIDI 360/44 com­putcr, a \'ideo Tape Com'encr (n'c), a VideoFilm Convcrter (\'FC), and a Video DataConsole (VDC).

The VTC is the de\·ice designed to com"ertanalog signals to digital form for use by the

:\BSTRACT: .,J lecll11ique by which lelet'ision pictures can be slored, processed,aud piciorilllly displayed by use of a digillli com puler syslem is designed 10 aidIhe pholoinlerpreler (user) in lhaguosing 1111 image ill all. oll-liue, mullipro­gramming, mlln-compuler colll'erslllionlll 111/l1 lime-slwring mode. The pholo­inlerpreler will be able to communicale wilh Ihe syslem vin II lighl pen, a key­board, and a Iypewriler 10 ask queslions lind make requesls, In addilion, Iheslored compuler progrnms may enhance /lIe image 10 a superior let'el of elllrilylind delail. The paper delails Ille Image Processing Syslem as '<f'ell as Ihe IIU­

merous splice and medical appliclliions.

ready for his request, the production run ",illbe interrupted and saved, and control will bereturned to the user with the conditions thatexisted previously. The multi-user mode \\'illpermit many terminals to be communicating"'ith the system under normal time-sharingconcepts.

/\. system si milar to that described is pres­en tly bei ng assem bled at Jet Propu IsionLaboratory, Pasadena, California, with theprogramming support of IBM/Federal Sys­tems Division, At JPL, the system \\'ill be uscdto process digitally television pictures ofspace bodies \\'hich were transmitted fromunmanned space \'ehicles. It \\'ill also be usedfor digi tal en ha ncemen t and processi ng ofmedical images, such as .c-ray photographs, indetermining tumors and other physicalanomalies.

The system described assumes an interac­tive CRT display console (tele\'ision) and ati me-sharing progra mmi ng system, However,the IPS as presently specified by JPL does notcontain these capabilities,

The next three sections detail each elemen tof the IPS and describe various applications ofthe system. The fourth section of this papercontains pictures that exemplify the outputof the system. These pictures were preparedunder the JPL system using an IB~I 360/44computer.

blAGE PROCESSI:\G \STDI

This section describes the various com­ponents of the I mage Processing System. The

digital computer. As they are recei\"ed fromthe spacecraft, the analog signals are ori­ginally stored on a magnetic tape that re­sembles a video tape. By use of the VTC, theanalog tape can be converted to a string ofdigital quantities, and the digital informationcan either be sent directly into the digitalcomputer system in real time, or stored on adigital magnetic tape for processing at a laterdate. The V'IC can com"ert the analog datain 6-bit or 8-bit quantities that representthe gray scale value at each particular tele\·i­sian picture point.

The VFC is the de\"ice that COI1\"ens the

EDWARD EFRO~

1060 PHOTOGRAMMETRIC E GINEERING

FIG. 1. IPS block diagram.

digital data output from the digital computerto analog form to generate the hard-copy pic­ture. The VFC can read the digital quantitiesdirectly from the digital computer system orfrom a digital magnetic tape. Subsequently, itconverts the signals to analog form to gen­erate the picture line by line on a cathode raytube which is automatically photographedunder a time exposure technique. The result isa 35 or 70-mm film from which hard-copyprints can be made. The VFC also has thecapabilities to scan a 35 or 70-mm negativeand create digital signals which can be fedinto the computer or stored on tape. Thisfeature enables the digi tal processing of pic­tures that are originally in hard-copy formsuch as chest x-ray photographs previouslytaken.

The VDC contains an interactive CRT (tele­vision) display which may operate in color orin black and white, and permits the user toview the complete image immediately afterthe digital computer generated it. Since a dis­play such as the VDC would be a useful addi­tion to the IPS, this paper assumes its avail­ability along with its required software.

FIG. 2. Video tape converter.

FIG. 3. Video film converter.

The VDC display is controlled by the digitalcomputer software and permits the user tocommunicate with the digital system bymeans of questions and requests via a lightpen and keyboard. The VDC with the relatedcontrolling software permits the photointer­preter (user) to manipulate digitally thetelevision picture in an on-line, man-com­pu tel' conversational fashion.

The digital computer is an IBM 360/44which utilizes tapes, disks, a printer, and atypewriter, and will be the master controllermanipulating all pictures and other data andhardware, as desired. I t will contain a mastersystem program (refer to the following sub­paragraph entitled Software) which will per­mit the user to call on all hardware and digitalcomputer programs by use of a specializedpicture processing language that may beeasily learned and used by individuals un­trained in programming.

Figure 1 shows a simplified block diagramof the above mentioned hardware units andtheir interrelationships.

Figures 2 through 4 show the VTC, the VFC,

and the IBM 360/44 computer as they are in­stalled at JPL, Pasadena, California. The VTC

FIG. 4. IBM 360/44 computer.

IMAGE PROCESSING BY DIGITAL SYSTEMS 1061

was designed and built by JPL. The VFC wasbuilt by Link/General Precision Corporationunder JPL'S specifications.

These photo!!,'raphs were taken by JPL andpresented to IBM for use in this paper.

SOFTWARE

The software package operates under the360/44 computer and consists of two pro­gram sets, the system programs and the func­tional programs. The following paragraphsdescribe each set individually.

System. The system programs are writtento permit the programmer and user to manip­ulate easily pictures and other data, as wellas hardware, by omitting the user perfor­mance of normally complex I/O and othercomplex computer programming tasks. Twosets of specialized language systems weredesigned, a problem programmer's languageand a user's language.

The problem programmer's language wasdesigned for use by the programmer whowishes to write additional picture manipula­tion routines (refer to the following sub­paragraph entitled "Functions"). This lan­guage or system consists of a group of sub­routines that may be called by the program­mer to handle all his complex television pic­ture I/O and all his input data transfers. Thisoption relieves the progmmmer of all I/Ooperations so that he can concentrate on themathematical or internal data manipulationprocedures; these may be written either inFortran or in assembly language.

The user's language was written to permitthe user or the individual untrained in pro­gramming to manipulate televi ion pictures,other data, and hardware through a simplifiedlanguage incorporating English words, with­out performing complex programming proce­d ures. This set of system programs translatesthe designed user language to meaningfulcomputer code and oversees the proper execu­tion of the code. The user language was de­signed with sufficient flexibility to allow theuser to execute several pre-written pictureprocessing routines (functions) on many dif­ferent pictures in a few simple statements.

Functions. The functional programs (func­tions) are the actual picture processing rou­tines used by the system. These are the digitalprograms that remove noise from a picture,correct for lens geometric distortion andcamera intensity errors, apply digital filtersto enhance the pictUt'es, compare previouslytaken chest x-ray photographs with the pres­ent ones to detect changes, etc.

Large numbers of simple and complex func-

tions may be written for image processingunder the system. For example, a function tosubdivide a picture into many smaller pic­tures is comparatively simple, whereas a func­tion to generate digitally a contour map froma surface photograph, or a function to per­form involved pattern recognition are quitecomplex.

ApPLICATIONS

There are many different types of applica­tions for a system such as the IPS. I n this re­port, however, only the space and medicalapplications which are of concern to JPL willbe discussed. Some of the following applica­tions require a display (such as the VDC) anda ti me-sharing software system.

The IPS will permit the space scientist torequest a CRT display of a particular pictureshowing the surface of a space body as takenfrom a space vehicle. He will then have theability to request that the system removefrom the picture all inaccuracies caused bythe camera lens (geometric distortion), bythe finite size of the camera scan beam, by thenonuniform vidicon brightness response ofthe television camera, or by any other tele­vision or transmission system noise.* He willthen obtain a much clearer, sharper, and ac­curate picture and be able to make decisionsabout the characteristics of the surface. Hecan then request by use of his light pen, key­board, typewriter, or cards, different projec­tions (views), enlargements, or even contourmaps of the specific areas of interest. Afterperforming various picture manipulations,he can request any desired number of hardcopies for subsequent analysis.

I t should also be remembered that thepicture processing described above occurs inan on-line, man-computer conversational mode,and that many terminals and production jobscan also be functioning.

The medical applications of IPS are alsoquite numerous. The IPS can aid the physicianin diagnosing his patient's ailment in thesame efficient and flexible manner describedabove by communicating with the systemvia a light pen, a keyboard, and a typewriterto ask questions and make requests.

The system will be capable of au tomaticallyreading pictorial medical data such as x-rayphotographs, extracting significant informa­tion, comparing it with previously stored dataand available knowledge, and pictoriallydisplaying on a CRT (television scope) the

* For details on digital correction techniques,see Reference 1.

1062 PHOTOGRA\1METRIC ENGINEERI~G

FIG. 5. Arm of Surveyor II [ extracting sample of lunar soil. The view on the left is BeforeGlare Filter, and the one on the right is .-lfter Glare Filter.

specific information that the physician re­quests. This philosophy was successfully at­tempted by JPL to detect chest tumorsthrough computer comparison of two x-raystaken at different times (Reference 2).

\\"ith the IPS, the computer possesses thecapability to detect the true signals concealedby the noise created by the x-ray photo­graphic process, and to display the x-ray pic­ture on a CRT at a level of clarity and detailthat could not normally be obtained. In ad­dition, the system could enlarge, manipulate,or further enhance any particular area spec­ified by the physician for his diagnosis, and/or possibly in the future, could use the com­puter as a specialist recommending the ap­propria te treatmen t to the physician. Thesystem could diagnose the case by using thepatient's past history and other similar casedata, as well as by referring to stored know­ledge generated by experts in that particularfield of medicine. The computer could per­form yarious pattern recognition operationsto locate features, and then perform measure­ments to determine the condition, such as themeasurements required to determine an en­larged heart. Therefore, wi th such a system, asingle physician could diagnose a case in ayery short period of time and \\'ithout fatigue,and obtain the same. if not better, resultsthan could be achie\'ed by a large group of~pe("iaiisis applying present techniques.

I LLU5TR ..HlO:\S

This section refers to the photographs thatillustrate the level of impro\'ement obtainableby use of the Image Processing System. The

following sets of pictures were prepared atJPL under their IPS.

The Frontispiece on page 1058 shows thelunar surface as taken by the Surveyor Ispacecraft. The picture entitled Before En­hancement was digitized, assembled on theIBM 360/44 computer, and played back on theVT'C to prod uce the hard copy. The pictureentitled Afler Enhancement underwent thesame process, except that a digital sine waveresponse filter was applied by a function pro­gram to emphasize the high-frequency signalsto restore their original strength. This processis used to compensate for the attenuation ofthe higher frequencies caused by the finitesize of the camera scan beam. The resultantpictu re shows the marked i m provementinsharpness and clarity.

Figure 5 shows the arm of SUr\'eyor IIIextracting a sample of the lunar soil. Thepicture entitled Before Clare Filter underwentthe same process as the picture entitled Be­fore Enhancement in the Frontispiece. Thesecond photograph entitled After Clare Filteralso underwent the same process as the firstpicture of this set, except that a digital filterwas applied to remove the sun's glare, whichis readily noticeable in the Before Glare Filterphotograph.

REFER E\'CES

1. I{. \'athan, "Digital Video-Data Handling,"'f'echnical Report 32-877, Jet Propulsiun Labora­lury, Pasadena, Calif., Jan. 5,1966.

2. R. H. Selzer, "Digital Computer Processing cfX-Ray Photographs," Technical Report 32­1028, Jet Propulsion Laboratory, Pasadena,Ca!if., Nov. 15, 1966.