upgrade of sonar analysis software for tiaps datacradpdf.drdc-rddc.gc.ca/pdfs/unc21/p521351.pdf ·...

Defence R&D Canada

DEFENCE DÉFENSE&

Upgrade of sonar analysis software for

TIAPS data

D.R. ChangEvans Computer Applications Ltd.

Evans Computer Applications Ltd.6424 Norwood St.Halifax, NS

Contract Number: W7707-021824/001/HAL

Contract Scientific Authority: S. Pecknold, 902-426-3100 x222

Contract Report

DRDC Atlantic CR 2003-041

February 2003

Copy No.________

Defence Research andDevelopment Canada

Recherche et développementpour la défense Canada

This page intentionally left blank.

Copy No:__________

Upgrade of sonar analysis software forTIAPS data

D. R. ChangEvans Computer Applications Ltd., 6424 Norwood St., Halifax, N.S.

Project manager: V. G. Evans, phone 902-422-6378

Contract number: W7707-021824/001/HAL

Contract scientific authority: S. Pecknold, Defence Scientist,phone 902-426-3100 extension 222

Defence R&D Canada – Atlantic

Contract Report

DRDC Atlantic CR 2003-041

February 2003

DRDC Atlantic CR 2003-041 i

Abstract

Sea trials of the towed integrated active passive sonar (TIAPS) provide data for the evaluation offuture possible sonar systems. The data produced by the TIAPS system was incompatible withsome existing Defence Research and Development Canada – Atlantic (DRDC Atlantic) sonaranalysis software. Therefore that software was upgraded to allow processing of the TIAPS data.

Résumé

Les essais en mer effectués par le sonar remorqué intégré actif et passif (TIAPS) fournissent desdonnées servant à évaluer les futurs systèmes sonars concevables. Les données obtenues duTIAPS ne pouvaient pas être entièrement analysées par les logiciels d’analyse sonar dont disposeRecherche et développement pour la défense Canada – Atlantique (RDDC Atlantique). Lelogiciel a donc été amélioré pour qu’il puisse traiter les données TIAPS.

ii DRDC Atlantic CR 2003-041


DRDC Atlantic CR 2003-041 iii

Executive summary

Introduction

The Towed Integrated Active Passive Sonar (TIAPS) project is a Technology Demonstrationproject designed to provide information that will be used to evaluate potential future sonarsystems. However, the data produced by TIAPS was incompatible with some existing DefenceResearch and Development Canada – Atlantic (DRDC Atlantic) sonar analysis software. Thisincompatibility arose because TIAPS uses a new data file format, and its data files can both belarger than previous files and contain more data channels.

Results

The convolution, beamforming, utility and display programs that are part of the analysis softwarepackage were updated to allow handling of data files greater than 2 GB in size. Various problemswith existing programs were fixed, notably with DISPX and AUDIODAT. The time seriesanalysis program PRODAT was changed so as to be able to handle files in the new data format.

Chang, D. R. 2003. Upgrade of sonar analysis software for TIAPS data. DRDC Atlantic CR 2003-041.

iv DRDC Atlantic CR 2003-041

Sommaire

Introduction

Le projet du sonar remorqué intégré actif et passif (TIAPS) est un projet de démonstration detechnologie conçu pour fournir des renseignements qui serviront à évaluer les systèmes sonarsconcevables. Toutefois, les données produites par le TIAPS ne pouvaient pas être entièrementanalysées par les logiciels d’analyse sonar de Recherche et développement pour la défenseCanada – Atlantique (RDDC Atlantique). En effet, le TIAPS utilise un nouveau format de fichierde données qui peuvent non seulement être plus grand que les anciens fichiers mais aussi contenirplus de pestes de données.

Résultats

Les programmes de convolution, de formation de faisceaux, de service et d’affichage quicomposent le progiciel d’analyse ont été amélioré pour pouvoir traiter des fichiers de donnéesplus grands que 2 Go. On a résolu divers problèmes posés par les programmes existants,notamment ceux présentés par le DISPX et l’AUDIODAT. Le programme d’analyse des sérieschronologiques, PRODAT, a été remplacé pour qu’on puisse traiter les fichiers du nouveauformat.

Chang, D. R. 2003. Upgrade of sonar analysis software for TIAPS data. DRDC Atlantic CR 2003-041.

DRDC Atlantic CR 2003-041 v

Table of contents

Abstract ..............................................................................................................................................i

Résumé ...............................................................................................................................................i

Executive summary..........................................................................................................................iii

Introduction ........................................................................................................................iii

Results.................................................................................................................................iii

Sommaire .........................................................................................................................................iv

Table of contents...............................................................................................................................v

List of figures ..................................................................................................................................vii

List of tables....................................................................................................................................vii

Support for TIAPS data files ............................................................................................................1

Updating the analysis software for TIAPS data...............................................................................3

Test and utility software......................................................................................................3

File utility software .............................................................................................................4

DISPX..................................................................................................................................8

Data convolution and matched filtering software ............................................................10

Heterodyning software ......................................................................................................23

REDUCEDAT...................................................................................................................28

BEAMFORM ....................................................................................................................28

RESAMPLE ......................................................................................................................32

AUDIODAT......................................................................................................................33

PRODAT ...........................................................................................................................33

PLTPWR ...........................................................................................................................41

PROPWR...........................................................................................................................42

Maximum array sizes......................................................................................................................44

MFILTR.............................................................................................................................44

vi DRDC Atlantic CR 2003-041

EXAM_PEAK...................................................................................................................45

Resolution of AUDIODAT problems ............................................................................................46

References .......................................................................................................................................48

List of symbols/abbreviations/acronyms/initialisms .....................................................................50

Distribution list ...............................................................................................................................51

DRDC Atlantic CR 2003-041 vii

List of figures

Figure 1. DISPX plot for a disk file .................................................................................................9

Figure 2. DISPX plot for a tape file ...............................................................................................10

Figure 3. Processed filtered data ....................................................................................................19

Figure 4. Statistics for processed filtered data ...............................................................................21

Figure 5. HETFIR input data..........................................................................................................24

Figure 6. HETFIR output data........................................................................................................26

Figure 8. RESAMPLE output data................................................................................................33

Figure 9. Real time series data........................................................................................................35

Figure 10. Complex time series data..............................................................................................36

Figure 11. Sample PLTPWR output...............................................................................................42

Figure 12. Sample PROPWR on-screen plot .................................................................................43

List of tables

Table 1. MFILTR arrays.................................................................................................................44

Table 2. EXAM_PEAK arrays .......................................................................................................45

viii DRDC Atlantic CR 2003-041


DRDC Atlantic CR 2003-041 1

Support for TIAPS data files

Originally, DRDC Atlantic acoustic data files were limited to at most 64 channels of data, dueto the way the data descriptor block or header of such files was defined (Caldwell [1]). Later anew header format was introduced to allow more channels (appendix A of Chang [2]). TIAPStime series data files use the latter format; such files containing 32-bit integer data aresometimes called DAT32 files. By comparison, time series files in the Caldwell format arecalled DAT files, and they usually contain 16-bit integer data. Although several sonar analysisprograms had already been modified to handle TIAPS files (Chang [3] and [4]), some otherprograms could not handle the new header format and/or 32-bit integer data.

Some TIAPS data files contain erroneous data in their headers. (These errors were revealedwhen the files were processed using program DISPX (Chang [5]), [6], [7], [8], and [3]).Sometimes the file start date was misaligned in the header. In other cases, the logical recordinformation in the header indicated that the file contained more data than it actually did. Toadjust for these errors, the sonar analysis programs were changed to correct misaligned dates,and to ignore the logical record data if they were at odds with the actual disk file size.

The combination of more channels, higher sampling rates, and higher media storage capacitiesmeans that TIAPS files can be larger than previous acoustic data files. In particular, files canbe larger than two gigabytes in size. The ability to process such files on a particular computerdepends on two factors:

• Whether the file system and computer languages support large files.

• Whether the application programs can handle large files.

Some sonar analysis programs could handle large files (Chang [3] and [4]), but they werelimited to the Microsoft (MS) Windows and SGI IRIX operating systems. It was required toadd support for large files under the Linux operating system

The sonar analysis programs are written in the C and FORTRAN programming languages.Recent versions of Linux allow large file handling via C. Support for large files underFORTRAN was reportedly available, but testing revealed that it did not work properly.Opening an existing large file (created via a C program) for read access under GNUFORTRAN (as in Red Hat Linux 8.0) could cause the contents of the file to be erased.Consequently the few FORTRAN programs that accessed acoustic data files via FORTRANinput/output (I/O) were changed to use C I/O to work around this problem.

Most of the analysis programs access acoustic data files via the MTAPEIO (magnetic tape)and DBIO (disk) I/O libraries (Chang [9], [6], and [8]). A few programs (PRODAT,PLTPWR, and PROPWR) instead use the so-called PORTIO functions for disk I/O. ThereforeMTAPEIO, DBIO and PORTIO were changed to add optional large file support under Linux.To allow Linux large file support when it is available, these libraries must be compiled underC with the macro _LARGEFILE64_SOURCE defined (i.e., -D_LARGEFILE64_SOURCE).

2 DRDC Atlantic CR 2003-041

Note that older versions of Linux (e.g., Red Hat Linux 7.0 and earlier) do not support largefiles.

In addition to adding optional large file support to MTAPEIO and DBIO, an attempt wasmade to add optional asynchronous I/O support (another recent Linux addition) to theselibraries. (Asynchronous I/O had been available in the VMS, MS Windows and IRIX versionsof the MTAPEIO and DBIO libraries.) However Linux asynchronous I/O did work with tapes,and was slower than synchronous I/O for disks (at least as used in program COPYDAT-Chang [10], [9], [6], [8], and [3]). Therefore asynchronous I/O has been disabled in the Linuxversion of MTAPEIO. Asynchronous I/O is available in Linux DBIO, but its use is notrecommended at present. (The only sonar analysis program which makes use of asynchronousdisk I/O is COPYDAT, and it does not currently use asynchronous I/O under Linux.) SinceLinux asynchronous I/O is done via functions in the librt library, Linux programs which useDBIO must now be linked with the librt library (i.e., -lrt) if librt exists.

Besides the modifications to the MTAPEIO, DBIO and PORTIO functions, some analysisprograms required changes to allow them to handle large files. The reason for this was that theprograms computed such things as the position within a data file and the amount of data to beprocessed using signed 32-bit integer arithmetic (i.e., integers £ 2.147483647e+9 in absolutevalue). As a result, such computations could be inaccurate when processing files containingtwo or more gigabytes. Therefore the programs were changed to do their file positioncomputations using greater precision arithmetic. A difficulty was that 64-integer support wasnot available in the version of Compaq Visual FORTRAN in use at DRDC Atlantic.Consequently programs that were to run under MS Windows were changed to do their fileposition computations using 64-bit floating-point arithmetic, which allows handling ofintegral values £ 9.0e+15 in absolute value. Program PRODAT (of which we were onlyconcerned with the Linux version) was modified to use 64-bit integer arithmetic for theposition computations, since GNU FORTRAN had 64-bit integer support.

Note that the MTAPEIO and DBIO libraries still use signed 32-bit integers to specify thephysical block numbers used when accessing disk or tape files. The PORTIO functions usebyte number when accessing disk files, but the programs using PORTIO still employ signed32-bit integers for physical block numbers. The block size for disk files is 512 bytes, so diskfiles of up to one thousand gigabytes (one terabyte) can be processed. Tape blocks are usuallyat least 1024 bytes long, and can be at most 65535 16-bit words or 131,070 bytes in length (alimitation imposed by the file header format). Therefore tape files of at most two thousand to262 thousand gigabytes (two to 262 terabytes) can be handled, depending on the block sizeused. Since the DLT IV tapes for TIAPS can hold at most 80 gigabytes of data (compressed),the above file capacities exceed the current TIAPS requirements. Modifying the sonaranalysis software to handle even larger tape and disk files by handling block numbers as 64-bit quantities is possible. However, it would be a lengthy task complicated by the lack of 64-bit integer support in some compilers. (The latter complication could be overcome for MSWindows systems if DRDC Atlantic updated their Compaq Visual FORTRAN compiler.)


Updating the analysis software for TIAPS data

The following sections summarize the changes made to the sonar analysis programs to bettersupport TIAPS data. (Note that many of these programs already had some support for thedata, and in some cases the only change made was to use the new Linux versions of DBIOand MTAPEIO.) To ensure that the software revisions did not affect the operation of theprograms, the same test data (files less than two gigabytes in size) was fed the both the oldand new versions of the programs. The resultant output files were compared to ensure thatthey agreed. The revised software was also tested with files of greater than two gigabytes insize; examples of such tests as run under Red Hat Linux 8.0 are given below.

The large time series files used to test the operation of the program were generated byprogram TEST_DBIO. The files had an "old" style header (Caldwell [1]) and contained twochannels of 32-bit integer data. The first channel contained the values 0, 2, 4, …, while thesecond channel contained 1, 3, 5, …. These data were not chosen as a realistic simulation ofsonar data, but rather to make it easy to check the operation of the programs with large files.

Test and utility software

Software developers use these programs to aid in testing the sonar analysis software. Theseprograms are in the CTEST and UTIL subdirectories, and are not present in DRDC AtlanticLinux software repository (Allen [19]).

TEST_DBIO

The C program TEST_DBIO is used to test the operation of the DBIO library. Thisprogram was changed so that it could generate large test time series files (greater thantwo gigabytes in size). Tests using this program indicated that the recent Linux DBIOadditions of large file support and asynchronous I/O worked properly.

TEST_MTAPEIO

The C program TEST_MTAPEIO is used to test the operation of the MTAPEIOlibrary. Tests using this program revealed that under Linux, large file support worked,but asynchronous I/O did not. Therefore support for asynchronous I/O was disabled inthe Linux version of MTAPEIO. The tests also revealed that the Linux tape driver hadsome difficulty moving backwards on a tape if it encountered an end of file mark.Some changes were made to the sonar analysis software to help compensate for thisproblem.

LISTDAT

The FORTRAN program LISTDAT lists and copies (possibly with user-specifiedmodifications) DRDC Atlantic standard binary disk files (Chang [11] and [12]). The


program was changed to handle larger disk files, and to compensate for some possibleerrors in TIAPS file headers (misaligned date and incorrect logical record data).Linux tests using LISTDAT indicated that there was a problem accessing large filesusing GNU FORTRAN I/O. Therefore the program was changed to instead use C I/Ovia the DBIO library.

DIFDAT

The FORTRAN program DIFDAT compares data from two DRDC Atlantic timeseries files (Chang [11], 12], and [13]). DIFDAT was changed to handle larger diskfiles. Like LISTDAT, it was changed to use C I/O via the DBIO library rather thanFORTRAN I/O.

File utility software

These programs list or copy standard DRDC Atlantic binary files.

COPYDAT

The C program COPYDAT copies user-selected data from an input binary file to anoutput binary file (Chang [10], [9], [6], [8], and [3]). Tests of Linux asynchronousdisk I/O using this program indicated that asynchronous I/O was slower thansynchronous I/O. Therefore asynchronous disk I/O was disabled in COPYDAT forLinux

• Here is an example of COPYDAT being used to copy a large file from disk totape. The input disk file, dcbig.dat, was created with program TEST_DBIO.Information about this input file obtained via the Linux command "ls -l":

-rw-r--r-- 1 chang chang 6144000512 Nov 18 14:26 dcbig.dat

A log of the COPYDAT run which copied part of dcbig.dat to tape:

Command> INIT ON

Command> MT_LABSIZE 1024

Command> MT_BLKSIZE 16384

Command> AMOUNT BL 4200000

Command> INFIL dcbig.dat

Opened input file dcbig.dat

Command> OFIL /dev/nst0/DCBIG2.DAT


Command> RUNInitializing /dev/nst0/ with volume name LINUX

Opened output file /dev/nst0/DCBIG2.DATCopying data ...131250 data block(s) written to /dev/nst0/DCBIG2.DAT

Command> LOG OFF

• Here is an example of COPYDAT being used to copy a large file from tape todisk. The input tape file, DCBIG2.DAT, was created by the above COPYDATrun. A log of the COPYDAT run used to copy DCBIG2.DAT from tape to disk:

Command> INFIL /dev/nst0/DCBIG2.DATFound file: DCBIG2.DAT

Opened input file /dev/nst0/DCBIG2.DAT

Command> OFIL bigdisk.dat

Command> RUN

Opened output file bigdisk.datCopying data ...4200000 data block(s) written to bigdisk.dat

Command> LOG OFF

Information about the output disk file bigdisk.dat as shown by "ls -l".

-rw-r--r-- 1 chang chang 2150400512 Nov 21 11:13 bigdisk.dat

DIRTAPE

The C program DIRTAPE produces a directory of all files of a tape (Chang [10], [9],[6], [7], [8], and [3]). The program was changed to correct misaligned dates read frominput file headers. Here is an example of DIRTAPE output for the large tape fileDCBIG2.DAT created in the aforementioned COPYDAT run:

Volume Name: LINUX

File Name Created Blocks Block Size File Size (bytes) (est. bytes)DCBIG2.DAT 20-NOV-2002 131251 16384 2.1504e+09

Start Date Start Time End Date End Time 10-AUG-81 16:07:47 10-AUG-81 18:27:47

----- DREA Data Descriptor for DCBIG2.DAT: -----

Integer Block:


8192 8192 -1 -1 1 4 2 1 1 1 1 1 0 0 0 0

Floating Point Block: 32000 0 2.32830644e-09 2.14748365e+09 1 -1 -1 -1

Byte Block: 1 0 2 0

ASCII Block: 10-AUG-81 16:07:47VOLTSA user message

---------------------------------------------------------------------

Estimated total size of DREA files: 2.1504e+09 bytes(Estimate excludes size of ANSI tape labels.)

DIRDISK

The C program DIRDISK produces a directory of user-specified disk files (Chang [7],[8], and [3]). The program was changed to correct misaligned dates read from inputfile headers. Below is an example of DIRDISK output for a large file, newbig.dat.First, information about the file as produced by "ls -l":

-rw-r--r-- 1 chang chang 3072000512 Nov 29 16:50 newbig.dat

DIRDISK output for this file:

../FUTIL/dirdisk -f newbig.dat newbig2.lis(Disk block size: 512 bytes)

File Name Date Blocks Bytes/home/chang/SW/CTEST/newbig.dat 29-NOV-2002 6000001 3.072e+09


----- DREA Data Descriptor for newbig.dat: -----

Integer Block: 256 -1 -1 -1 1 4 2 1 1 1 1 1 0 0 0 0

Floating Point Block: 32000 0 2.32830644e-09 2.14748365e+09 1 -1 -1 -1


Byte Block: 1 0 2 0

ASCII Block: 10-AUG-81 16:07:47VOLTSA user message

---------------------------------------------------------------------

Total size of 1 DREA file(s): 3.072e+09 bytes

DUMPDAT

The C program DUMPDAT lists the contents of standard DRDC Atlantic binary files(Chang [13] and [8]). Following is an example of DUMPDAT output for a large file,newbig.dat. First, information about the file as produced by "ls -l":


A listing of the header, the first data block, and the last data block for the above file asproduced by DUMPDAT:

DATA DESCRIPTOR FOR newbig.dat:[Header occupies 1 block(s).]

*** INTEGER BLOCK (ILABEL) ***(____1_____2______3______4______5______6______7______8______9_____10) 256 -1 -1 -1 1 4 2 1 1 1 1 1 0 0 0 0

*** FLOATING-POINT BLOCK (FLABEL) ***(____________1______________2______________3______________4) 3.200000e+04 0.000000e+00 2.328306e-09 2.147484e+09 1.000000e+00 -1.000000e+00 -1.000000e+00 -1.000000e+00

*** BYTE BLOCK (BLABEL) ***(____1_____2______3______4______5______6______7______8______9_____10) 1 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 . . . 0 0 0 0 0 0 0 0

*** ASCII BLOCK (ALABEL) ***

ALABEL (50A1 format):(12345678901234567890123456789012345678901234567890) 10-AUG-81 16:07:47 VOLTS A user message

Frame: Channel(s)


_____________1_____________2 1: 0 1 2: 2 3 . . . 63: 124 125 64: 126 127

Frame: Channel(s) _____________1_____________2 383999937: 767999872 767999873 383999938: 767999874 767999875 . . . 383999999: 767999996 767999997 384000000: 767999998 767999999

DISPX

The FORTRAN program DISPX selectively displays data from a time series data file (Chang[5]), [6], [7], [8], and [3]). The program can also selectively output the data to another timeseries file, a sound file, a SAPLOT file, or a set of MAGIC files. DISPX was modified tohandle large files, and to allow for errors in the header of input files (misaligned dates orincorrect logical record data). Moreover, DISPX was changed so that the headers of anyoutput time series data file it generated were of the same type as the headers of thecorresponding input file. (Previously DISPX output "old" style headers (Caldwell [1])whenever possible.)


Here is a sample DISPX plot for a large disk file, dcbig.dat:

Figure 1. DISPX plot for a disk file


Here is a sample DISPX plot for a large tape file, DCBIG2.DAT:

Figure 2. DISPX plot for a tape file

Data convolution and matched filtering software

This software is designed to perform convolution or correlation of time-multiplexed, multi-channel, time series data files with separate, arbitrary waveforms. Optionally, someprocessing of the output of the convolution/correlation is then performed (Chang [14], [15],[2], [12], [6], and [3]). The programs mentioned below were modified to correct for possiblemisaligned dates in any input TIAPS time series files.

TIMCON

The FORTRAN program TIMCON performs time domain convolution or correlationof time series data with separate, arbitrary waveforms (kernels) (i.e., the time seriesdata are filtered). Optionally some processing of the filtered data is then performed.TIMCON was modified to handle 32-bit integer data in the time series or kernel files.The program was also changed to process large data files. Here is an example of


TIMCON being used to process a large time series file, newbig.dat. First, informationabout this input file as produced by "ls -l":


• Here is the report file output from the TIMCON run:

Time series input: newbig.dat

Control file: TIMCON.CON Kernel file: ONEPT.KER Operation type (0=convolution, 1=correlation): 0 Start data point: 1 End data point (<0 for all): -1 Remove zero fill from end of kernel data (Y/N): N Kernel normalization (1=SQRT(N)*RMS, 2=N*RMS, 3=N*MS, 4=FSAMPLE):1 FFT length: 256 No. of points to average: 1 No. of points of overlap: 0

Data Kernel .MFR Gain .MFP Gain Channel Channel (dB) (dB) 1 0 0.000 0.000 2 0 0.000 0.000

Accessing ONEPT.KER END OF FILE ENCOUNTERED. Using 1 real kernel(s) of length 1

Kernel Normalization Channel Factor (dB) 0 0.000

Accessing newbig.dat

Raw filter output: newbigtim.MFR

Working ... END OF FILE ENCOUNTERED.

No. of data blocks output to .MFR file= 6000000

Output Integer Channel Overflows 1 0. 2 0.

CPU sec for convolution/correlation = 125.5600

• Information regarding the output file as shown by "ls -l":


-rw-r--r-- 1 chang chang 3072000512 Nov 30 23:29newbigtim.MFR

FRECON

The FORTRAN program FRECON performs frequency domain convolution orcorrelation of time series data with separate, arbitrary waveforms (kernels) (i.e., thetime series data are filtered). Optionally some processing of the filtered data is thenperformed. FRECON was modified to handle 32-bit integer data in the time series orkernel files. The program was also changed to process large data files. Here is anexample of FRECON being used to process a large time series file, newbig.dat. First,information about this input file as produced by "ls -l":


• Here is the report file output from the FRECON run:


Control file: TIMCON.CON Kernel file: ONEPT.KER Operation type (0=convolution, 1=correlation): 0 Start data point: 1 End data point (<0 for all): -1 Remove zero fill from end of kernel data (Y/N): N Kernel normalization (1=SQRT(N)*RMS, 2=N*RMS, 3=N*MS, 4=FSAMPLE):1 FFT length: 256 No. of points to average: 1 No. of points of overlap: 0

Data Kernel .MFR Gain .MFP Gain Channel Channel (dB) (dB) 1 0 0.000 0.000 2 0 0.000 0.000

Accessing ONEPT.KER END OF FILE ENCOUNTERED. Using 1 real kernel(s) of length 1



Raw filter output: newbigfre.MFR


No. of data blocks output to .MFR file= 6000000


Output Integer Channel Overflows 1 0. 2 0.


• Information regarding the output file as per "ls -l":

-rw-r--r-- 1 chang chang 3072000512 Dec 1 16:08newbigfre.MFR

PROCON

The FORTRAN program PROCON processes the raw filtered data produced byprogram TIMCON or FRECON. PROCON was modified to handle 32-bit integerdata, and to process large data files. Here is an example of PROCON being used toprocess a large file, newbigfre.MFR, as produced in the above FRECON test run.

• Here is the report file output from the PROCON run:

Raw filtered input: newbigfre.MFR

Control file: PROCON2.CON Start data point: 1 End data point (<0 for all): -1 No. of points to average: 1 No. of points of overlap: 0

Channel .MFP Gain (dB) 1 0.000 2 0.000

Accessing newbigfre.MFR

Processed filter output: newbigfre.MFP


No. of data blocks output to .MFP file= 6000000

CPU sec for processing = 72.71000

• Information regarding the output file as per "ls -l":

-rw-r--r-- 1 chang chang 3072000512 Dec 2 12:29newbigfre.MFP


MFILTR

The FORTRAN program MFILTR performs matched filtering of time series datawith separate, arbitrary waveforms (kernels). The filtering consists of correlating orconvolving the time series data with the kernels and/or the Hilbert transform of thekernels. MFILTR was modified to allow larger files to be processed.

• Here is an example of MFILTR being used to process data from a large disk file,dcbig.dat, with output going to disk. First, information regarding the input file per"ls -l":

-rw-r--r-- 1 chang chang 6144000512 Nov 18 14:26 dcbig.dat

The MFILTR report file for this run:

Time series input: dcbig.dat

Control file: MFILTR.CON Kernel file: ONEPT.KER Operation type (0=convolution, 1=correlation): 0 Type of filter output (1=normal, 2=quadrature, 3=normal+quad):1 Start data point: 1 End data point (<0 for all): 268800000 Remove zero fill from end of kernel data (Y/N): N Kernel normalization (1=SQRT(N)*RMS, 2=N*RMS, 3=N*MS, 4=FSAMPLE):1 FFT length: 512 Subkernel length (if <=0, set to kernel length): 1 No. of points of overlap between subkernels: 0

Data Kernel .MFP Gain Channel Channel (dB) 1 0 0.000 2 0 0.000

Accessing ONEPT.KER END OF FILE ENCOUNTERED. Read 1 real kernel(s) of length 1 No. of subkernels per kernel: 1


Accessing dcbig.dat

.MFP_NORM output: dcbig.MFP_NORM

Working ...

No. of data blocks output to .MFP_NORM file= 4200000



Information regarding the output file for the run as obtained via "ls -l":

-rw-r--r-- 1 chang chang 2150400512 Nov 22 15:08dcbig.MFP_NORM

• Here is an example of MFILTR being used to process data from a large disk file,newbig.dat, with output going to tape. Information about newbig.dat obtainedfrom "ls -l":


Here is the report file for the MFILTR run:


Control file: MFILTR2.CON Kernel file: ONEPT.KER Operation type (0=convolution, 1=correlation): 0 Type of filter output (1=normal, 2=quadrature, 3=normal+quad):1 Start data point: 1 End data point (<0 for all): -1 Remove zero fill from end of kernel data (Y/N): N Kernel normalization (1=SQRT(N)*RMS, 2=N*RMS, 3=N*MS, 4=FSAMPLE):1 FFT length: 512 Subkernel length (if <=0, set to kernel length): 1 No. of points of overlap between subkernels: 0





.MFP_NORM output: /dev/nst0/newbigt.MFP_NORM REWINDING TAPE FOR INITIALIZATION!





Information regarding the output tape file as obtained via program DIRTAPE:

Volume Name: RT11A

File Name Created Blocks Block Size File Size (bytes) (est. bytes)NEWBIGT.MFP_NORM 29-NOV-2002 375001 8192 3.072e+09


----- DREA Data Descriptor for NEWBIGT.MFP_NORM: -----

Integer Block: 4096 512 -1 -1 2 4 2 1 1 1 2 1 0 -999 0 0

Floating Point Block: 32000 0 1 -1 1 1 0 1

Byte Block: 1 0 2 0

ASCII Block: 10-AUG-81 16:07:47VOLTS**2A user message Report: newbigt.REP

---------------------------------------------------------------------

Estimated total size of DREA files: 3.072e+09 bytes(Estimate excludes size of ANSI tape labels.)

• Here is an example of MFILTR being used to process data from a large tape file,DCBIG2.DAT. (This tape file is the same one created above in the COPYDATsample run.) Here is the report file for the MFILTR run:

Time series input: /dev/nst0/DCBIG2.DAT

Control file: MFILTR2.CON Kernel file: ONEPT.KER Operation type (0=convolution, 1=correlation): 0 Type of filter output (1=normal, 2=quadrature, 3=normal+quad):1 Start data point: 1 End data point (<0 for all): -1


Remove zero fill from end of kernel data (Y/N): N Kernel normalization (1=SQRT(N)*RMS, 2=N*RMS, 3=N*MS, 4=FSAMPLE):1 FFT length: 512 Subkernel length (if <=0, set to kernel length): 1 No. of points of overlap between subkernels: 0




Accessing /dev/nst0/DCBIG2.DAT FOUND FILE: DCBIG2.DAT

.MFP_NORM output: DCBIG2.MFP_NORM




Information regarding the output file for the run as obtained via "ls -l":

-rw-r--r-- 1 chang chang 2150400512 Nov 22 16:28DCBIG2.MFP_NORM

MFPLOT

The FORTRAN program MFPLOT copies processed filtered data from a binary .MFPfile to an ASCII graphics file for plotting. MFPLOT was changed to handle largerinput files. Here is the ASCII SAPLOT file produced by MFPLOT for the first 50points of the large file DCBIG2.MFP_NORM produced above by MFILTR:

! SAPLOT file: 'DCBIG2.SAP'FORMAT 1LABEL 7DCBIG2.MFP_NORMLABEL 310-AUG-81 16:07:47LABEL 1


Time (sec)RANGE 2 -200.0 -100.0LABEL 2dB re 1 volt**2LEGEND 2Ch 1Ch 2CURVE ! curve # 1 0.000000 -267.0506 0.3125000E-04 -166.6391 . . . 0.1500000E-02 -133.0138 0.1531250E-02 -132.8347CURVE ! curve # 2 0.000000 -172.6594 0.3125000E-04 -163.1171 . . . 0.1500000E-02 -132.9238 0.1531250E-02 -132.7465


Here is a plot of the above SAPLOT file:

Figure 3. Processed filtered data

MFSTAT

The FORTRAN program MFSTAT calculates statistics across channels of processedconvolution/correlation data. MFSTAT was modified to handle larger input files.Following is the report file output from MFSTAT when the large fileDCBIG2.MFP_NORM was processed:

Input control file: MFSTAT.CON

CONTROL FILE DATA: ------------------ Statistics type (1=mean, 2=st. dev., 3=both): 3


Start data point: 1 End data point (<0 for all): -1

No. of channels in group 1 = 2 Channel names for group 1: 1 2 Mean for group 1 will be output to channel 0 St. dev. for group 1 will be output to channel 1

No. of channels in group 2 = 0

PROCESSING: ----------- Accessing DCBIG2.MFP_NORM Statistics output: DCBIG2_STATS.MFP_NORM Working ... END OF FILE ENCOUNTERED. No. of data blocks output to .MFP file= 4200000 CPU sec for processing = 56.24000

• Information regarding the output file as produced by "ls -l":

-rw-r--r-- 1 chang chang 2150400512 Nov 25 13:44DCBIG2_STATS.MFP_NORM


A plot of the first 50 points of the above output file:

Figure 4. Statistics for processed filtered data

MFPEAK

The FORTRAN program MFPEAK finds peaks in the processed filtered data.MFPEAK was modified to handle larger input files. Here are the contents of theoutput auxiliary file DCBIG2.AUX_NORM produced when part of the large fileDCBIG2.MFP_NORM was processed by MFPEAK (no peaks were found for thesecond channel):

Control file: MFPEAK6.CON Start data point (0 for auto-select): 1 End data point (0 or -1 for auto-select, -2 for EOF): 20000 Zero fill threshold (dB): -350.0000 Segment length: 25


Straighten threshold for noise level curve: 0.5000000 Threshold above noise for peaks: 2.000000 Minimum number of points in a peak: 5 Minimum number of points between peaks: 6 Segment noise estimation method (1=centre of fitted line, 2=mean of point estimates, 3=median of point estimates) : 1

Accessing DCBIG2.MFP_NORM/CHANNEL= 1/THRESHOLD (dB)= 11.53269/NOISE (sec, dB) 0.3750000E-03 -151.6210 0.5843750E-02 -121.2082 . . . 0.6238125 -80.63451 0.6245937 -80.62364/PEAKS (sec, sec) 0.9687500E-03 0.2375000E-02/CHANNEL= 2/THRESHOLD (dB)= 11.52975/NOISE (sec, dB) 0.3750000E-03 -147.2459 0.4281250E-02 -123.8845 . . . 0.6238125 -80.63430 0.6245937 -80.62343/PEAKS (sec, sec)

EXAM_PEAK

The FORTRAN program EXAM_PEAK determines some information about thepeaks located by program MFPEAK. EXAM_PEAK was modified to handle largerinput files. However since EXAM_PEAK reads the entire contents of the inputprocessed filtered data file into memory at once, the amount of data EXAM_PEAKcan process is limited by the memory of the computer.

MFNORM

The FORTRAN program MFNORM normalizes matched filter data by subtractingthe estimated noise level in dB from the filter data in dB. MFNORM was modified tohandle larger input files. MFNORM processes files of partly demultiplexed data(multiplex length > 1) produced by the matched filtering programs MF2 and MF3,rather than the multiplexed data (multiplex length = 1) files created by programMFILTR. We do not have access to MF2 and MF3, which are confidential programswritten by Vance Crowe of DRDC Atlantic.


PEAKFILE

The C program PEAKFILE creates a binary file containing the peaks from a matchedfilter file. PEAKFILE was modified to handle larger input files. However,PEAKFILE stores peak locations in its output file as sample numbers (32-bitintegers), which limits the amount of data it can process.

Heterodyning software

HETFIR

The FORTRAN program HETFIR heterodynes, filters, and/or decimates time seriesdata (Chang [16], [12], [6], [8], and [4]). HETFIR was changed to handle larger inputfiles. HETFIR was also modified to correct for possible misaligned dates in any inputTIAPS time series files. Here is an example of HETFIR being used to process a largefile, newbig.dat. Information about newbig.dat as obtained via "ls -l":



• A plot of the contents of this HETFIR input file:

Figure 5. HETFIR input data

• A log of the HETFIR run:

../HETFIR/hetfir

TIME SERIES DATA FILTERING AND HETERODYNING PROGRAM (Dec. 02, 2002)

Name of the input time series file (LOWFRQ.DAT ?) ? newbig.dat

The input file starts at 16:07:47 Do you wish to begin processing at some other time (N ?) ?

512 bytes per block in the input file. Indicate the amount of input data to process. Method (1=blocks, 2=duration, 3=stop time) (1 ?) ?


No. of blocks to process (-1 for all) (-1 ?) ?

Channel names: 1 2

How many channels to process (-1 for all) (-1 ?) ?

10-Jul-92/Coefficient file must have gain factor! Consult V. Crowe Name of input filter file (mda3.cof ?) ? MDA1.COF

Decimation factor= 25 No. of filter coefficients= 250 Filter gain will assumed to be one (0 dB) after applying acorrection factor of -96.32018 dB to the FIR coefficients. Additional gain (dBs) (0. ?) ?

Combined multiplicative constant is 0.1527534E-04 Heterodyne frequency (0.0 for no heterodyning) (0. ?) ?

Type of real output: 1 = Same as input. 2 = 32-bit floating-point. Type of real output (1 ?) ?

Input file is newbig.dat

Name of the output time series file (newbig.het ?) ?

AP_On = F Dummy AP Select routine entered

Processing ... END OF FILE ENCOUNTERED. 240000 data block(s) output. Filtering/heterodyning took 116.8000 CPU sec.

Do you want to process another file (N ?) ?

• Information about the output file:

-rw-r--r-- 1 chang chang 122880512 Dec 2 17:10 newbig.het


A plot of the contents of the output file:

Figure 6. HETFIR output data

UNHET

The C program UNHET interpolates and/or unheterodynes time series data. UNHETwas modified to allow large files to be handled. Following is a sample run of UNHETin which it whose input file, newbig.het, was created by the above HETFIR run. Theoutput report file from the UNHET run:

Input file: newbig.hetFirst point to interpolate between: 1Last point to interpolate between: -1No. of channels to process: 2Scale factor for output data: 1Interpolation factor: 25Filter coefficients file: COEF8.DATHeterodyning frequency (Hz): 0Output data type (1=16-bit integer, 2=16-bit floating-point, 3=32-bit integer, 4=32-bit floating-point): 3Output time series file: newbig.unhet


Elapsed processor time = 174.08 sec.

• Information about the output file newbig.unhet:

-rw-r--r-- 1 chang chang 3071999488 Dec 4 17:41newbig.unhet


A plot of the contents of newbig.unhet:

REDUCEDAT

The C program REDUCEDAT reduces the amount of data in a time series file by selecting arepresentative portion of the data (Chang [2] and [12]). REDUCEDAT was modified tohandle large data files. . However, REDUCEDAT stores data information in its output indexfile as sample numbers (32-bit integers), which limits the amount of data it can process.

BEAMFORM

The FORTRAN program BEAMFORM performs time domain beamforming of hydrophonearray data (Chang [4]). BEAMFORM was modified to handle larger input files. Following is asample run of BEAMFORM. Program BFSETUP (Chang [4]) is run before BEAMFORM todefine the hydrophone array configuration for BEAMFORM.

• Defining the hydrophone array configuration via BFSETUP:

./bfsetup PROGRAM BFSETUP (Oct. 27, 2001) Sets the beamformer configuration for an array.


Do you want to log your responses (N ?) ? y Enter file name to log responses (beam_config.log ?) ?

PROGRAM OPTIONS: ================ 1 = Define line array 2 = Define general array 3 = Save configuration 4 = Restore configuration from file 5 = Compute beam patterns 6 = Quit program

Which option (1 ?) ? 1

DEFINE A LINE ARRAY: ====================

Number of interpolation coefficients (8 ?) ? Coefficient file name (ics_coef8.dat ?) ? No. of coefficients in file= 8 The sampling rate (Hz) (2048. ?) ? 32000 The speed of sound in the water (m/s) (1500. ?) ? Basic inter-element spacing (m) (2. ?) ? Number of hydrophones (128 ?) ? 2 Number of beams (64 ?) ? 2

Set hydrophone positions: -------------------------

This program understands a line array composed of up to four sub-groups with the shortest spacing first and the longer spacings trailing. Also the number of channels halves with each sub-group.

How many sub-groups (1 ?) ? Basic spacing= 2. Group spacing will be = 2.

Choose beams: -------------

Choose a single channel for the last beam or 0 (0 ?) ?

Beam angle options are: 0 = Evenly spaced in angles 1 = Evenly spaced in sine of angles 2 = Each angle is unique

Select beam angle option (0-2) (0 ?) ? Of 2 beams in the fan, where do you want the Starting angle (-90. ?) ? Ending angle (90. ?) ?


Choose shading: ---------------

Shading functions are: 1=square, 2=Hanning, 3=Hamming, 4=Taylor, 5=triangular

Shading function (3 ?) ?

Choose gains: ------------- Gain steps or exponents must be between 0 and 15, where gain step 0 is -6 dB, 1 is -12 dB, etc.

Choose gain step or exponent for beams (0 ?) ?


Which option (3 ?) ? 3 Do you want to save the data on disk (Y ?) ? Initial file size= 33 blocks. Disk file name (beam_config.bin ?) ? Finished saving beamformer configuration.


Which option (3 ?) ? 6STOP End of array definition beamformer program statement executed

• The input hydrophone data file to be processed by BEAMFORM:

[chang@FRODO BEAMFORM]$ ls -l ../CTEST/newbig.dat-rw-r--r-- 1 chang chang 3072000512 Nov 29 16:50../CTEST/newbig.dat

• The BEAMFORM run:


[chang@FRODO BEAMFORM]$ ./beamform

TIME DOMAIN BEAMFORMING PROGRAM (Dec. 05, 2002)

Name of the input time series file (Q to quit) (beamin.dat ?) ? ../CTEST/newbig.dat

Input binary beamformer configuration file (Q to quit)(beam_config.bin ?) ? No. of hydrophones= 2, No. of beams= 2

No. of interpolation coefficients= 8, Sampling frequency (Hz)=32000

Finished retrieving configuration data.

The input file starts at 16:07:47 Do you wish to begin processing at some other time (N ?) ?

512 bytes per block in the input file. Indicate the amount of input data to process. Method (1=blocks, 2=duration, 3=stop time) (1 ?) ?

No. of blocks to process (-1 for all) (-1 ?) ?

How many channels to process (-1 for all) (-1 ?) ?

Type of output: 1 = Same as input. 2 = 32-bit floating-point. Type of real output (1 ?) ?

Input time series file is ../CTEST/newbig.dat

Name of the output time series file (newbigb.dat ?) ?

Processing ... END OF FILE ENCOUNTERED. Beamforming took 393.2300 CPU sec.

Name of the input time series file (Q to quit) (../CTEST/newbig.dat?) ? q

• The output file produced by BEAMFORM:

[chang@FRODO BEAMFORM]$ ls -l newbigb.dat-rw-r--r-- 1 chang chang 3072000000 Dec 5 21:12 newbigb.dat


RESAMPLE

The C program RESAMPLE resamples real (versus complex) time series data, therebygenerating data that can have a different sampling frequency. RESAMPLE was modified tohandle larger input files. A bug was also corrected in RESAMPLE whereby files thatcontained only one channel weren't being processed properly. Following is an example ofRESAMPLE being used to process a large file, newbig.dat.

• Information regarding the input file per "ls -l":


• The output report file from the RESAMPLE run:

Input file: newbig.datStart processing 0 second(s) from beginning of file.Process to end of input file.

Process 2 channel(s):1,2

Input sampling frequency: 32000 HzOutput sampling frequency: 25600 HzResampling method: linear interpolationOutput time series file: newbig_R.dat

Elapsed processor time = 267.54 sec.

• Information about the output time series file:

-rw-r--r-- 1 chang chang 2457600512 Dec 3 17:41newbig_R.dat


A plot of the contents of newbig_R.dat:

Figure 8. RESAMPLE output data

AUDIODAT

The C program AUDIODAT outputs one or two channel real (versus complex) time seriesdata to a sound card. AUDIODAT was modified to handle larger input time series files.

PRODAT

PRODAT is a FORTRAN program for performing spectral analysis of time series data(Farrell [22], Murray [23], Maxwell [24], Chang [8]). The Linux version of PRODAT wasmodified to handle larger input files, and files with the new data descriptor (see appendix A of{Chang] 2). The program was also modified so that it could process 32-bit integer time seriesdata.

Another change to the program was to modify Linux PRODAT so that it could read spectralfiles in non-native format. Further, PRODAT now writes output spectral files in big endianformat, consistent with an earlier DRDC decision that PRODAT output files be in big endianformat. Since programs PLTPWR and PROPWR cannot handle input spectral files which


have use the new header format, PRODAT uses the old header format (Caldwell [1]) foroutput spectral files with £ 64 channels. When using the old header format, PRODAT limitsoutput channel names to the range 0 to 255; this is done by usingmod(abs(input_channel_name),256) as the output channel name.

A further change was made regarding the optional channel description (.DES) file input by theprogram. This file formerly contained the following data in each line of the file: channelnumber, RF channel, sonobuoy type, and demux channel in the FORTRANFORMAT(I3,1X,I2,1X,A4,1X,A3). To allow for larger RF channel values, PRODAT nowcan also accept these data in FORMAT(I3,1X,I3,1X,A4,1X,A3), and .DES files written by theprogram are in this latter format.

The PRODAT Front End (PFE), a separate program which provides a GUI interface toPRODAT, was not changed due to time limitations. Consequently, the PFE cannot be usedwhen processing files that contain a new data descriptor.

The maximum number of channels that can be processed by Linux PRODAT is currently setat 128. (The computer we were given access to by the Scientific Authority only had 128Mbytes of memory, which did not permit processing more than 128 channels in PRODATunder Red Hat Linux 8.0.) To allow more channels to be processed in PRODAT, one mustchange two files. In the file parameters.common, the parameter MAX_CHAN should be set tothe maximum number of channels. The parameter BLABEL_SIZE in this file should be set toan integer equal to 256+N*512, where N is an integer ≥ 0 and BLABEL_SIZE ≥2*MAX_CHAN. One must also change the definition of BLABEL_SIZE in the file drea.h tobe consistent with its definition in parameters.common.

Our main concern in testing the revised PRODAT program was to ensure that it could stillprocess old .DAT time series files containing 16-bit integer data, as well as new .DAT32 filescontaining 32-bit integer data. Using program DUMDAT (Chang [15]), we generated two.DAT files with old type headers (Caldwell [1]) containing 16-bit integer data. The first filecontained four channels of real data: a 100 Hz sine signal, a 100 Hz cosine signal, a 200 Hzsine signal, and a 200 Hz cosine signal. The second file contained four channels of complexdata whose real components were the same as in the first file, and whose imaginarycomponents were zero. (Note that program DUMDAT exists only in the UTIL directory onthe VMS platform.)


Figure 9. Real time series data


Figure 10. Complex time series data

To provide test files in little endian and big endian IEEE formats, the above VAX format fileswere copied to IEEE format files using program COPYDAT (Chang [10], [9], [6], [8], and[3]). Further, program LISTDAT was used to produce copies of the files which had the newtype data descriptor and 32-bit integer data. Thus there were six real test data files and sixcomplex test data files, corresponding to the three possible files formats (VAX, little IEEEand big IEEE) and either 16-bit integer time series data (with a .DAT type file descriptor) or32-bit integer time series data (with a .DAT32 file descriptor).

The HMOD command in PRODAT allows one to change values in the header of a file. To testthis feature, the various test files were opened, and had their header data changed andrewritten via PRODAT. The files were then reread to ensure that the headers had beenmodified properly.

To test the spectral analysis calculations, we subjected all six test files to the followinganalysis using the revised version of PRODAT. Power spectra were computed for channels 1to 4 and were output to a .pwr file. Greyscale data for the four channels were output to a .gryfile. Cross spectra and coherence were computed for channels (1,2) and (3,4) and were outputto .crs and .coh files respectively. Default settings were used for all PRODAT analysisparameters.To provide a reference set of .pwr, .gry, .crs and .coh files we subjected the IEEE


little endian 16-bit real and complex data files to the same analysis, but using the previousversion of PRODAT. The files output by the revised PRODAT were compared with thesereference files, and the results were found to match in all cases. For comparing the .pwr, .crsand .coh files we used program DIFDAT (Chang [11], [12], and [13]). Since DIFDAT couldnot compare byte data, to compare the .gry data we listed the contents of the files via programDUMPDAT (Chang [13] and [8]). We then compared the .gry file listings via the system diffcommand.

Following is some sample output from a run of program DIFDAT for the real data. Thereal4linc.* files were generated by the revised PRODAT from little endian IEEE 16-bitinteger time series data. The ../prodat_old/real4lin.* files were produced by the previousversion of PRODAT from little endian IEEE 16-bit integer time series data. Note thatDIFDAT is primarily designed to compare data from multiplexed time series files. In order touse it to compare the .pwr, .crs and .coh files, we asked DIFDAT to treat such files as if theycontained only one channel. The "max. allowable" difference values are just parameters usedto tell DIFDAT when it should count two points as being different. (Any differing points areprinted by DIFDAT.)

real4linc.pwr format appears to be big endian IEEE ../prodat_old/real4lin.pwr format appears to be native Comparing channel number 1 from real4linc.pwr with channel number 1 from ../prodat_old/real4lin.pwr 1st file contains real data. 2nd file contains real data. Starting comparison at beginning of files. Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000 Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of real4linc.pwr

Stopped at point 8192. of first file and point8192. of second. 8192. points compared and 0. weredifferent. Max. per cent difference between non-zero points= 0.000000 RMS per cent difference between non-zero points= 0.000000 Max. absolute difference between non-zero points= 0.000000 Max. absolute difference between zero points= 0.000000

real4linc.crs format appears to be big endian IEEE ../prodat_old/real4lin.crs format appears to be native Comparing channel number 1 from real4linc.crs with channel number 1 from ../prodat_old/real4lin.crs 1st file contains complex data. 2nd file contains complex data. Starting comparison at beginning of files.


Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000 Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of real4linc.crs


real4linc.coh format appears to be big endian IEEE ../prodat_old/real4lin.coh format appears to be native Comparing channel number 1 from real4linc.coh with channel number 1 from ../prodat_old/real4lin.coh 1st file contains complex data. 2nd file contains complex data. Starting comparison at beginning of files. Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000 Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of real4linc.coh


Following is some sample output from a run of program DIFDAT for complex data. Thenucomplex4big.* files were generated by the revised PRODAT from big endian IEEE 32-bitinteger time series data. The ../prodat_old/complex4lin.* files were produced by the previousversion of PRODAT from little endian IEEE 16-bit integer time series data.

nucomplex4big.pwr format appears to be big endian IEEE ../prodat_old/complex4lin.pwr format appears to be native Comparing channel number 1 from nucomplex4big.pwr with channel number 1 from ../prodat_old/complex4lin.pwr 1st file contains real data.


2nd file contains real data. Starting comparison at beginning of files. Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000 Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of nucomplex4big.pwr


nucomplex4big.crs format appears to be big endian IEEE ../prodat_old/complex4lin.crs format appears to be native Comparing channel number 1 from nucomplex4big.crs with channel number 1 from ../prodat_old/complex4lin.crs 1st file contains complex data. 2nd file contains complex data. Starting comparison at beginning of files. Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000 Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of nucomplex4big.crs


nucomplex4big.coh format appears to be big endian IEEE ../prodat_old/complex4lin.coh format appears to be native Comparing channel number 1 from nucomplex4big.coh with channel number 1 from ../prodat_old/complex4lin.coh 1st file contains complex data. 2nd file contains complex data. Starting comparison at beginning of files. Max. allowable per cent difference between non-zero points=0.1000000E-01 Max. allowable absolute difference between zero points= 0.000000


Beginning comparison at point 1. of 1st file andpoint 1. of 2nd. EOF/error in record 66 of nucomplex4big.coh


To test that the revised version of PRODAT could handle large data files, we used as the inputthe large file newbig.dat mentioned above in the section on progam DUMPDAT. Informationregarding newbig.dat as obtained by "ls -l":


We computed power spectra for channels 1 and 2 as follows. (We chose not to process theentire time series file, as this would have taken too much time.)

** CURRENT PRODAT PARAMETERS ** -------------------------------------------------------------------- Input File : newbig.dat Output File : newbig.pwr File start : 16:07:47 Analysis start : 16:07:47 Time per FFT : 00:00:00.032 # FFTs Averaged : 10 Averaging Time : 00:00:00.176 Set Center Sep. : 00:00:00.160 Total Time : 00:53:20.016 # Sets To Do : 20000 Data Pts./ FFT : 1024 Zero Pad Pts. : 0 Window Type : SQUARE % Overlap : 50.00 Strt/Stp Bins : 1/ 512 Min/Max Freqs : 0.0/15968.8 Mean Removal : OFF Bins Averaged : 1 Resolution (Hz): 31.25000 Calibration : OFF Normalization : SPLIT_WINDOW Greyscaling : OFF Input Channels : 1-2 Output Channels : 1-2 Input H/P : 1-2 Output H/P : 1-2 Clutter Tool : OFF Clutter File : newbig.clt Display Size : 89% X 89% KD Sin Theta : OFF --------------------------------------------------------------------

Information regarding the output file per "ls -l":

-rw-rw-r-- 1 chang chang 81920512 Jan 28 18:54 newbig.pwr

A plot of the some of the data from newbig.pwr appears in the following section on PLTPWR.


PLTPWR

PLTPWR is a FORTRAN program for producing on-screen and hardcopy plots of spectraldata (Farrell [25]; Chang [5], [7], and [8]). PLTPWR was modified to handle larger inputspectral data files. However, the limit on the number of frames PLTPWR can process in asingle run was not changed. Due to time limitations, PLTPWR was not changed to handledata files that have a new header (appendix A of Chang [2]).

Here is a sample run of PLTPWR using as input the file newbig.pwr mentioned in the abovesection on PRODAT:

** PLTPWR STATUS **

===================================================================== Input File : newbig.pwr Output File : newbig.ps001 File start : 16:07:47.000 File stop : 17:01:06.855 Input frames : 1-20000 Output frames : 1-4 Input channels : 1-2 Output channels : 1-2 Input H/P : 1-2 Output H/P : 1-2 Averages (Input): 10 Averages (Output): 10 Compress. factor: 1 Compress. method : NONE Preference : FRAMES Curves per plot : 4 Offset : 10.000 Pts. Per Curve : 512 X Axis Type : LINEAR Y Axis Type : LINEAR X Min/Max : AUTOSCALED Y Min/Max : AUTOSCALED Hcopy Interval : 1 (File) Display Time : INFINITE List Processing : OFF Onscreen Plot : ON Normalization : SPLIT_WINDOW Grey/Gram/Cal : OFF/OFF/OFF

=====================================================================


Figure 11. Sample PLTPWR output

PROPWR

PROPWR is a FORTRAN program for processing power spectrum files (Farrell [26]; Chang[5] and [8]). PROPWR was modified to handle larger input spectral data files. However, thelimit on the number of frames PROPWR can process in a single run was not changed. Due totime limitations, PROPWR was not changed to handle data files that have a new header (seeappendix A of Chang [2]).

Here is a sample run of PROPWR using as input the file newbig.pwr mentioned in the abovesection on PRODAT:

** CURRENT PROPWR PARAMETERS ** -------------------------------------------------------------------- Input file : newbig.pwr Input Channels : 1-2 Input H/Ps : 1-2 File start : 16:07:47 File stop : 17:01:06.840 Frames in file : 20000 Min. Frequency : 0.0 Max. Frequency : 15968.75 -------------------------------------------------------------------- Output Channels : 1-2 Output H/Ps : 1-2 Selected Frames : 10000-10100 Offset : 10.00


Processing start : 16:34:27.680 Processing stop : 16:34:43.680 1/3 octave cent : 10.0 # 1/3 oct. bnds : 24 1/3 Oct. Process.: ON 1/3 Octave File :newbig.octsap Band Time Series : OFF Line Tracking : OFF Trend removal : OFF Calibration : OFF Line Calibration: OFF -------------------------------------------------------------------

Figure 12. Sample PROPWR on-screen plot


Maximum array sizes

The Scientific Authority requested that the current maximum array sizes be tabulated for theFORTRAN programs MFILTR (Chang [14, [15], [2], [6], and [3]) and EXAM_PEAK(Chang[14, [15], [2], and [6]).

MFILTR

Table 1. MFILTR arrays

TYPE AND NAME SIZE DESCRIPTION

REAL DATBUF (2097089) Time series data

REAL ENDBUF (3145728) .MFP_END data buffer

REAL FFTBUF (32768) Fourier coefficient data

REAL FILBUF (32768) Filtered data

REAL FL2BUF (32768) Filtered data

REAL HILBUF (32770, 32) Hilbert transformed kernel data

INTEGER*2 INBUF (2101248) Input buffer

REAL KERBUF (32768, 32) Kernel data

REAL NORBUF (2097152) .MFP_NORM data buffer


EXAM_PEAK

The sizes of arrays DATBUF, XTIME and YDB were increased as follows per the ScientificAuthority's instructions.

Table 2. EXAM_PEAK arrays

TYPE AND NAME SIZE DESCRIPTION

REAL DATBUF (1600000, 8) Input data in volts**2

INTEGER*2 INBUF (16384) Input buffer

REAL XNOISE (4000) Times for noise data

REAL XTIME (1600000) Times (sec.) of data

REAL YDB (160000) Data in dB

REAL YNOISE (4000) Noise level data in dB


Resolution of AUDIODAT problems

Program AUDIODAT outputs one or two channel real (versus complex) time series data to asound card (Chang [4]). We investigated problems with sound card output in the Linuxversion of the program. One general problem we found (not Linux specific) was thatAUDIODAT truncated its output when playing back data from a time series data file. Thisproblem was due to output buffers not being flushed in AUDIODAT; this situation wascorrected. Another general problem corrected was that time series files that contained onlyone channel weren't being processed properly. Moreover, we fine tuned some programconstants to reduce clicks on audio playback. We also updated the PortAudio sound libraryused by AUDIODAT to version 18, and included any patches available as of December 12,2002 (Burk [17]).

Testing of AUDIODAT was done primarily by using as input two TPULSE control files,song.con and song2.con, as well as the corresponding .DAT time series files generated byprogram TPULSE (Chang [20] and [21]).

Here are the contents of the file song.con. (song2.con was the same except that it specifiedtwo channels, rather than one.)

11025 ! Sampling frequency (Hz)10 ! File length (s)50 55 27 350 ! Noise filter rolloff, attenuation, min. & max. length1 ! No. of channels5 ! No. of pulsesCW 440 440 HAN 1.80 0 0 0 0CW 493.88 493.88 HAN 1.80 0 2 0 0CW 523.55 523.55 HAN 1.80 0 4 0 0CW 440 440 HAN 1.80 0 6 0 0CW 523.55 523.55 HAN 1.80 0 8 0 0

The Linux problems with AUDIODAT were of two types:

• The program output the message ">> PortAudio initialization error: Host error". Thiscould be due to one or both of the following situations:

a. Some other process is using the sound card digital signal processor (DSP)/dev/dsp. To see if a process is using this device, one can use the "fuser"command as in "/sbin/fuser -v /dev/dsp". If one is using the KDE desktopenvironment, the artsd daemon may be controlling access to the sound card. Onecan change the behaviour of artsd via the "KDE Control Center / Sound / SoundServer" menu item. Decreasing the time before artsd autosuspends when idle, or


else not running artsd, will reduce or eliminate artsd interference withAUDIODAT.

b. The user did not have read/write access to the /dev/dsp device. One can check theaccess permissions of this device by the command "ls -l /dev/dsp". To allowanyone to access this device, one can enter the following command while loggedin as root: "chmod ugo+rw /dev/dsp". Note however that Linux may possiblyretain this change in access permissions only during current login session.

• The program output the message "Pa_SetupDeviceFormat: HW does not support xchannels. >> Error opening PortAudio stream: Host error". (The actual message was "HWdoes not support 2 channels", which has since been corrected to be "HW does not support1 channels.") The cause of this message was that the sound card driver did not supportmono output under Linux. Therefore AUDIODAT was changed so that if one asks formono output, the program issues the message "Warning the sound card may not supportmono output. Do you wish to use stereo output?" If one answers yes, AUDIODAT willconvert the output to stereo form by sending the same signal to both left and right outputs.

Note that AUDIODAT does not adjust the balance or volume of the sound output. Suchadjustments can be done under MS Windows by the MS Volume Control program, or underLinux via program aumix.

As in the case of the MS Windows version of AUDIODAT, one may hear occasional pops orclicks during the Linux playback of data from a time series file. This is probably a latencyissue (Burk [18]). These pops and clicks may be reduced if one increases the size of theconstant MIN_RAW_BUF_SEC defined in the source files audiodat.c and audiodat_proc.cAlternatively, reducing the sampling frequency and/or using mono versus stereo playbackmay help matters.


References

1. Caldwell, D. A. (1987). A standard for the DREA data descriptor block. (DREA technicalcommunication 87/302). Defence Research Establishment Atlantic.

2. Chang, D. R. (1995). User's manual for active sonar acoustic data matched filtering andclustering software. (Unpublished DREA contractor report). Defence ResearchEstablishment Atlantic.

3. Chang, D. R. (2001). New and updated underwater sound data analysis software.(Unpublished DRDC Atlantic contract report). Defence Research and DevelopmentCanada Atlantic.

4. Chang, D. R. (2001). Time domain audio and beamforming analysis software.(Unpublished DRDC Atlantic contract report). Defence Research and DevelopmentCanada Atlantic.

5. Chang, D. R. (2001). Final report and user's manual for modifications to PLTPWR, DISP,DISPX and PROPWR. (DREA-CR-2001-106). Defence Research Establishment Atlantic.

6. Chang, D. R. (1999). Final report and user's manual for changes and updates to acousticdata manipulation software. (Unpublished DREA contractor report). Defence ResearchEstablishment Atlantic.

7. Chang, D. R. (2000). Final report and user's manual for changes and additions to acousticanalysis software. (Unpublished DREA contractor report). Defence ResearchEstablishment Atlantic.

8. Chang, D. R. (2001). Final report on the port of signal processing software to Linux.(DREA-CR-2001-108). Defence Research Establishment Atlantic.

9. Chang, D. R. (1999). Technical manual for file utility software. (Unpublished DREAcontractor report). Defence Research Establishment Atlantic.

10. Chang, D. R. (1999). User's manual for file utility software. (Unpublished DREAcontractor report). Defence Research Establishment Atlantic.

11. Chang, D. R. (1993). Final summary report for the data convolution project. (UnpublishedDREA contractor report). Defence Research Establishment Atlantic.

12. Chang, D. R. (1995). Final summary report for the active sonar acoustic data matchedfiltering and clustering software. (Unpublished DREA contractor report). DefenceResearch Establishment Atlantic.

13. Chang, D. R. (2000). User's manual for programs to interpolate between spectral data, tolist data files, and to merge IMPACT data. (Unpublished DREA contractor report).Defence Research Establishment Atlantic.


14. Chang, D. R. (1996). User's manual for the data convolution software. (UnpublishedDREA contractor report). Defence Research Establishment Atlantic.

15. Chang, D. R. (1993). Final summary report for the data convolution project. (UnpublishedDREA contractor report). Defence Research Establishment Atlantic.

16. Chang, D. R. (1994). User's manual for data filtering and heterodyning software. (DREA-CR-94-446). Defence Research Establishment Atlantic.

17. Burk, P. (December 3, 2002). PortAudio - an open-source cross-platform audio API.(Online). SoftSynth.com. http://www.portaudio.com (December 12, 2002).

18. Burk, P. (December 3, 2002). PortAudio latency. (Online). SoftSyn.com.http://www.portaudio.com/docs/latency.html (December 12, 2002).

19. Allen, C. (2002). Recommended standard for software management and control using theConcurrent Versions System (DRDC Atlantic TM 2002-057). Defence Research andDevelopment Canada - Atlantic.

20. Chang, D. R. (1997). Program TPULSE for generating simulated acoustic data. (DREACR/97/407). Defence Research Establishment Atlantic.

21. Chang, D. R. (2001). Program TPULSE for generating simulated acoustic data.(Unpublished DREA contractor report). Defence Research Establishment Atlantic.

22. Farrell, J. B. and Heard, G. J. (1992). PRODAT: a versatile tool for spectral analysis anddisplay. (DREA technical communication 92/309). Defence Research EstablishmentAtlantic.

23. Murray, K. (1997). PRODAT II development. (DREA CR/97/403). Defence ResearchEstablishment Atlantic.

24. Maxwell, C. (1999). PRODAT enhancement and repair. (DREA CR 1999/048). DefenceResearch Establishment Atlantic.

25. Farrell, J. B. (1992). PLTPWR: a versatile tool for greyscale and power spectral display.(DREA technical communication 92/306). Defence Research Establishment Atlantic.

26. Farrell, J. B. (1993). PROPWR: a versatile tool for 1/3 octave processing and narrowbandline tracking. (DREA technical communication 93/303). Defence Research EstablishmentAtlantic.


List of symbols/abbreviations/acronyms/initialisms

DRDC Defence Research and Development Canada

DREA Defence Research Establishment Atlantic

DSP Digital signal processor

I/O Input/output

MS Microsoft

PFE PRODAT Front End

TIAPS Towed integrated active passive sonar


Distribution list

Document No.: DRDC Atlantic CR 2003-041

LIST PART 1: CONTROLLED BY DRDC Atlantic LIBRARY

3 DRDC Atlantic LIBRARY FILE COPIES3 DRDC Atlantic LIBRARY (SPARES)

1 S. Pecknold1 J. Theriault1 B. Maranda1 I. Fraser

____________________10 TOTAL LIST PART 1

--------------------------------------------------------------------------------------------------------

LIST PART 2: DISTRIBUTED BY DRDKIM 3

1 NDHQ/ CRAD/ DRDKIM 3 (scanned and stored as black & white image, low resolution - laser reprints available on request )* Full mailing address must be supplied for units other than NDHQ

___________________1 TOTAL LIST PART 2

11 TOTAL COPIES REQUIRED

--------------------------------------------------------------------------------------------------------

upgrade of sonar analysis software for tiaps datacradpdf.drdc-rddc.gc.ca/pdfs/unc21/p521351.pdf ·...

Documents