technical manual - europa · technical manual . application-to-application data exchange toolkit...

THE STATEL PROJECT Application-to-Application Data Exchange Toolkit

Technical Manual

A P P L I C A T I O N - T O - A P P L I C A T I O N D A T A E X C H A N G E T O O L K I T

Technical Manual

2003-2006 Eurostat Written by: S A Eeckels Updated by: European Dynamics SA

Table of ContentsINTRODUCTION 1 Background 1 Concepts 2 Independence 2 The Virtual File System 4 The STATEL Nick Names 4 The Protocol Data Unit 5 Virtual File System: Advantages 5 Application Program Interface 5 Modularity 8 Security and reliability 9 Security 9 Reliability 10 New Features 10 IMPLEMENTATION 13 Topology 13 The API environment 14 The Transfer Module Environment 15 The STATEL Gateway 15 The STATEL HUB 16 System structure 18 STATEL Sequence Numbers 18 STATEL messages 19 The STATEL Virtual File System (VFS) 19 File types 19 File contents 19 VFS Logical Structure 20 File attributes 20 VFS Internal Structure 21 Locking incomplete files 21 Communication Principles 22 Requirements 22

Protocol overview 22 Tuning the PDU size 23 Segmenting large files 23 Delayed transmission 23 Encryption and compression 24 Key Agreement Principles 24 Payload Encryption 24 Rewriting the PDU 24 Decrypting and expanding the PDU 25 CONFIGURATION PRINCIPLES 27 Configuring the STATEL API 27 Fundamentals 27 Defining the STATEL universe 29 Configuration file expansion rules 29 Configuring the STATEL Nick Names 30 Setting the system defaults 31 Transit Directories 31 Retransmit delay 31 Remotely executable commands 31 Logging options 31 Configuring the tuples 31 Defining the Transfer Agents 32 Setting Tuple Parameters 33 Entering Protocol Lines 34 Configuring the Transfer Modules 34 ftp 34 Description 34 Support strategy 34

HTTP 35 Description 35 Support strategy 35

SMTP e-mail systems 35 Description 35 Support strategy 36 Configuration at Eurostat 37 DESIGNING STATEL APPLICATIONS39 Language bindings 39 STATEL for Perl 39 Overview of Perl/STATEL commands 39 STATEL for Tcl/Tk 40 Overview of Tcl/STATEL commands 40 Programming example 41 STATEL Command Line Utilities 42 VFS utilities 42 vfscopy, vfsmove and vfscat 43 vfsdir 43 vfsrm 43 vfsmkdir 44 vfsrmdir 44 vfsmesg 44 vfsulock 44 vfspart 44 vfspass 45 vfspredir 45 vfsextract 45 vfsreset 45

Implementation 47 Developing with STATEL 47 General concepts 47 Monitoring the data exchange process 47 Keeping application files in the VFS 48 Data security 48 CAPTURE - a data capture program 48 Data Flow 49 The client program 49 Directories and files 50 Processing 50 The server program 52 STATEL PDU SPECIFICATIONS 55 Syntax 55 Protocol lines 57

Internal STATEL Commands 57 ISC.ID 57 ISC.TR 58 ISC.MS 58 ISC.RX 58 ISC.MP 58 ISC.RS 58 ISC.KE 59 ISC.EN 59 File Management Requests 59 FMR.CF 59 FMR.DF 60 FMR.UF 60 FMR.RL 60 FMR.FS 61 FMR.CX, FMR.UX, FMR.DX and FMR.RX 61 Remote Command Execution 61 RCE.LI 61 RCE.SR 61 RCE.CP 61 RCE.EX 61 Limiting the size of a PDU 62 Interpreting the STATEL PDU 62 The pduaccess API 62 PDU enumeration 63 Opening and closing PDU files 63 Setting the open modes 63 Accessing PDU components 64 Service functions 66 LOGGING FEATURES 67 Logging of urgent events 67 Introduction 67 Activating the logging feature 67 Size monitoring and backups 68 Format of the log entries 68 Description of the log entries 68 Support for logging of remote errors 69 Introduction 69 Activating the feature 69 On the remote system 69 On the local system 70 Size monitoring and backups 70

Format of the log entries 70 Description of the log entries 70 Support for the transmission of reports70 Activating the feature 71 Size monitoring and backups 71 Format of the log entries 71 BUILDING STATEL 72 Introduction 72 The STATEL Core 72 STATEL for Unix 73 Providing OpenSSL 73 Building process overview 73 Compiling zlib 75

STATEL Core for Windows 75 Perl language bindings 75 Java language bindings 75 Tcl/Tk language bindings 75 STATEL Command Line Utilities 75

List of figures Figure 1: STATEL draws the line ..................................................................................... 3 Figure 2: Logical structure of STATEL ............................................................................ 4 Figure 3: Relationship between the STATEL primitives .................................................. 7 Figure 4: STATEL Modularity........................................................................................... 9 Figure 5: STATEL Component Distribution .................................................................... 14 Figure 6: The STATEL Hub ............................................................................................ 18 Figure 7: STATEL Virtual File System - logical structure............................................... 20 Figure 8: STATEL Virtual File System - Implementation................................................ 21 Figure 9: Layout of a STATEL Installation..................................................................... 28 Figure 10: The STATEL installation at Eurostat ............................................................ 37 Figure 11: CAPTURE - Data Flow Diagram..................................................................... 49

I N T R O D U C T I O N

1

Introduction Situating STATEL, new features

Background Telecommunication is, without doubt, one of the most dynamic fields in information processing. We’re now in the broadband era, and Peer-to-Peer file “sharing” applications have the big name record labels running for cover (or more accurately, applying flawed CD protection methods and suing grandfathers). Web-based applications are widely deemed a panacea (and aren’t, as all claimed panacea).

It’s therefore slightly puzzling to see that the tools proposed for the purpose of data transfer between (business) applications are FTP and email (SMTP), and a raft of “proprietary” applications that mostly aim to improve on FTP, but seem incapable of cutting loose of FTP’s fundamental design approach: move a file to and from a remote computer system, and nothing else.

All these data exchange applications using telecommunications have their own idiosyncratic requirements, are heavily dependent on the availability of a specific facility, network, or piece of hardware or software. The same problems are solved over and over again, but in very specific, ad-hoc ways. In addition, many "standard" protocols are oriented towards solving specific problems, and more or less force applications to use a particular approach, determined more by the design and limitations of the protocol than the requirements of the application. Indeed, when looking at telecommunications as experienced by the users, we see quite often people acting as interfaces between computer programs, a truly bizarre situation in this 21st century. This is where STATEL offers a solution.

The goal of the STATEL data exchange primitives is to provide uniform telecommunication facilities between remote applications, and to shield these applications from modifications in the computing environment. The key word here is applications, because it identifies STATEL clearly as a service layer, which, once installed, should be as invisible as possible.

Chapter

1


2

After defining STATEL as a service to applications, it is clear that it can not be an application itself, however sophisticated; to be effective, it has to be integrated with the applications using the data. So first and foremost (and to use a currently fashionable acronym), STATEL is an API (Application Program Interface), a set of function calls allowing the application developers to formalize the data exchange with the applications of the remote party, without worrying about details of transfer, protocol, operating system and hardware. This API hides the differences between the (often conflicting) approaches to telecommunications (e.g. connected or store-and-forward protocols), restricts access to only those files required by the applications, harmonises the file naming conventions of the various operating systems, and offers the possibility to dispense with the requirement to grant personalised access to communicating computers or networks. STATEL is easy to learn by application programmers, and presents an intuitive, recognisable programming paradigm.

Second, STATEL is also a standard. The STATEL Protocol Data Units use a well-defined syntax, and could be generated by other systems than the implementation provided by Eurostat. One of the nice things of the STATEL PDU standard is that it isn’t an overly ambitious standard with many options; a basic implementation is easily realised.

Third, STATEL is an “architecture”, through the provision of a number of infrastructure tools that enable the interconnection of STATEL installations, based on the realities of today’s networks with their DMZs (De-Militarised Zones), firewalls, proxies, and restrictive security policies.

Concepts Independence By designing an application with STATEL, the application designer gains significant independence from

1. The telecommunication facilities.

2. The local system and network topology.

3. The remote host file naming conventions and restrictions.

4. The remote host user naming conventions and restrictions.

5. The remote host program execution facilities.

6. The system clock settings of the local and remote hosts.

The STATEL conventions replace the host machine and protocol conventions, and this allows the development of applications that need not be aware of all the different environments (i.e. hardware, operating systems, telecommunication


3

facilities, protocols, rules, local requirements, etc.). We're not performing any magic, as these differences are real (and, if I might be so bold, desirable), we're just building a buffer between the application and the telecommunications reality. Changes to this reality affect STATEL, not the applications. This, effectively, draws a line between the application and the telecommunications world (see Figure 1).

Figure 1: STATEL draws the line

The details of the transfer and the specifics of the host computers are hidden by the STATEL layer. If the remote party decides to move its operations from a VMS machine1 to a UNIX machine, the local user doesn't need to change her applications. She doesn't even have to change her STATEL configuration, provided the remote party keeps his usual address and communications method, and installs STATEL on the new computer. This feature greatly eases the administration of systems communicating with large numbers of correspondents.

Another way to represent the added value of STATEL is shown in Figure 2. Logically, the applications dialogue with each other, independent and unaware of the activities of STATEL or the underlying transport mechanism. Interestingly, the STATEL API layer itself is unaware of the differences in transport mechanisms, and uses a well defined API to communicate with the Transfer Modules (TMs) that perform the actual dialog with the transport mechanism. This architecture makes it easy to add support for a new transport mechanism.

1 Assuming there would be a STATEL implementation for VMS, which there isn’t, unfortunately.


4

Figure 2: Logical structure of STATEL

The stabilisation of developments is another important advantage from the introduction of this clear-cut separation of duties between the application and the data transmission service. All applications using STATEL immediately benefit from the introduction of support for a new protocol, and application developers can eliminate the telecommunications bug-bear from their considerations.

One important point should be made here: STATEL does not attempt to convert files. More and more, the meaning of the bytes used by applications has moved away from the basic conventions embodied in ASCII and EBCDIC, and their extensions. When applications are designed to cooperate through STATEL, they should agree on the meaning of the data they transmit; STATEL will faithfully reproduce at the other site whatever 8-bit bytes (octets) it is given to transfer.

The Virtual File System The key concept of the STATEL primitives is the provision of a mirrored Virtual File System or VFS. This VFS presents a unified view of the files, and only those files, that have been exchanged between the two parties. In other words, the application(s) at both ends of the link can place files in the VFS, and these files will be accessible to either party, independent of the party that placed the files in the VFS. Read performance is optimised through the presence of a local copy, and in addition to the data, the VFS structures are also used to provide applications with metadata related to the actual transfer process.

The STATEL Nick Names The names of the STATEL parties are defined in the configuration files. Every STATEL environment has its own logical name or STATEL Nick Name (SNN).


5

When communicating with another STATEL party, the combination of the two SNNs (local and remote) uniquely identifies the VFS that will be accessed.

The Protocol Data Unit All the files in the VFS are present in the STATEL area on the local storage facility, and can be accessed as easily as local files. Because the access takes place through the STATEL API, a set of instructions can be generated that reflect the changes made to the information in the VFS. This set of instructions (which we'll call a Protocol Data Unit (PDU), to re-use a consecrated term) can be transferred to the remote machine by whatever means available, where it is used to effect exactly the same changes to the copy of the VFS stored there2. The PDU is a single file (data unit), and contains all the information required to allow an orderly progression from one VFS state to another. PDUs carry a unique sequence number, and hence can be processed in the correct order. Gaps in the sequence cause processing to be halted, and a request for the missing PDUs to be sent to the remote party.

Virtual File System: Advantages STATEL maintains a local copy of the VFS, and uses the PDUs obtained from the remote party to integrate the changes made by the remote party into the local copy. Both parties can access all the data in the VFS, with the ease and speed of local file access. This approach all but eliminates the traditional concept of file transfer. Applications can be developed which communicate directly through the VFS, and logically dispense with the intermediate stages of writing a local file.

The VFS also solves a number of compatibility problems between Operating Systems. The file naming conventions of operating systems differ widely, and the many file transfer and networking packages have adopted varying methods to find common ground between the systems. Rather than using the conventions of the host OS, STATEL uses its own file name conventions, and the implementation uses a map between its name (currently an ASCII string of up to 64 characters) and a set of simple OS names. As a result, applications can rely on the STATEL file name as a common convention, without needing to adapt the name to a potentially unknown remote Operating System.

Application Program Interface The distinguishing feature is the provision of a UNIX-style API for this VFS. This makes it easy to develop applications which use the VFS and the local file system in a compatible fashion. It must again be noted that STATEL has no

2 To aid comprehension, one can compare the way STATEL operates with two people playing correspondence chess; both have a chess board and chess men (not necessarily identical in shape and colour), which represent the VFS. They exchange their moves using a piece of paper and a conventional notation (eg E2-E4), the PDU. When the parties receive the PDUs, they effect changes to the chess board, so that it accurately represents the new situation. As with all comparisons, the match is not perfect, but if you keep the chess game in mind while reading this manual, you will find it a lot easier to understand the mechanisms behind STATEL.


6

notion of file type, and leaves the applications to ensure that they understand each other's files. Table 1 gives an overview of the STATEL API:

Primitive Description General functions STATEL_INITIALIZE Initialises the STATEL session STATEL_EXEC Executes a STATEL command on the remote host STATEL_TERMINATE Terminates the STATEL session. STATEL_RESET Reset the sequence numbers with or without reinitialisation of

the VFS StatelBuildSNNList Call a user callback for every remote SNN, using the name of

the remote party, and the tuple directory as parameters StatelGetLocalSNN Return the local SNN, can be used even when

STATEL_INITIALIZE has not yet been called. StatelUnlock Used to remove the lock from an SNN. uGetBase Return the STATEL Base Directory Data transfers outside the VFS STATEL_PUTMESSAGE Transfers a message; no trace of the message is kept locally, and

no acknowledgment of its reception is sent by the remote party. STATEL_GETMESSAGE Accepts a message from the remote party. VFS primitives STATEL_STARTVFS Start VFS operations. STATEL_OPEN Create or open a file in the VFS. STATEL_STAT, STATEL_ISTAT, STATEL_FSTAT

Obtain file status information.

STATEL_READ Read a sequence of bytes from a file. STATEL_WRITE Write a sequence of bytes to a file. STATEL_SEEK Seek to a position in a file. STATEL_TELL Return the current file pointer. STATEL_CLOSE Close a file previously opened by STATEL_OPEN. STATEL_UNLINK Remove a file from the VFS STATEL_RENAME Rename a file in the VFS STATEL_MKDIR Create a sub-directory in the VFS STATEL_CHDIR Change the current working VFS directory STATEL_RMDIR Remove a subdirectory from the VFS STATEL_FCNTL Control VFS file attributes Service routines STATEL_ERRORCODE Return the value of the STATEL error variable. Introduced in

version 3.1 to support environments such as MS-Windows which do not support global variables in shared libraries.

STATEL_ERROR Return a STATEL or system error message based on the current value of the STATEL error variable. Supports multi-lingual output (LANG environment variable).

uError Print an error message, based on the STATEL error or the system error table. Supports multi-lingual output (LANG environment variable). Obsolescent. Not available in STATEL for Windows.

Table 1: Overview of the STATEL API

The STATEL_INITIALIZE routine is used to identify the party, and to do whatever is required to bring the contents of the STATEL environment up to date. When this routine returns, STATEL_GETMESSAGE can be used to retrieve messages sent by the remote party, and STATEL_PUTMESSAGE to send messages. If the VFS is to be accessed, the STATEL_STARTVFS function must be called, after which STATEL_OPEN can be used to access a file in the VFS. Opening a


7

non-existing file for WRITE access will automatically create it. A successful STATEL_OPEN call returns a file handle, allowing the file to be read and written to using STATEL_READ and STATEL_WRITE. STATEL_SEEK is used to position the file pointer in the VFS file. STATEL_CLOSE ends the processing of the specified VFS file. Up to 7 VFS files can be open simultaneously. STATEL_TERMINATE closes the activities for this STATEL tuple, and transfers the possible PDU to the transfer service (depending on the configuration, it might launch (run) a transfer program). Figure 3 depicts the relationships between the primitives.

Figure 3: Relationship between the STATEL primitives

The manual pages describing the primitives are included in an appendix to this document. The interface offered here is the well-known, UNIX like file system interface. This interface is sufficiently powerful to support sophisticated data transfer facilities efficiently, and has been incorporated in all modern 'C' language developments.

Application

STATEL_INITIALIZE()

STATEL_STARTVFS() STATEL_PUTMESSAGE() STATEL_GETMESSAGE()

STATEL_EXEC()

STATEL_OPEN() STATEL_READ() STATEL_WRITE() STATEL_CLOSE() STATEL_SEEK() STATEL_EXEC()

STATEL_TERMINATE()

Application


8

main(argc, argv) { FILE *source; int destination; char buffer[512]; int length; if (STATEL_INITIALIZE(argv[1]) != NULL) { STATEL_STARTVFS(); source = fopen(argv[2], "r"); destination = STATEL_OPEN(argv[3], WRITE, ""); while ((length = fread(buffer, 1, 512, source)) > 0) { STATEL_WRITE(destination, buffer, (long)length)); } fclose(source); STATEL_CLOSE(destination); STATEL_TERMINATE(); } }

Listing 1: Basic STATEL send file program

The example program in Listing 1 shows clearly how easy it is to implement a basic file transfer program using the STATEL primitives. Because of the close relationship between the STATEL primitives and the normal file access facilities of modern Operating Systems, it becomes easy to design applications that seamlessly integrate data exchange in their structure. This is certainly easier, and more natural, than separating the file transfer from the processing of the data, as is the case with mail based APIs (MAPI/CMC). Less obvious advantages are that it becomes very easy to send only updates to existing files in the VFS, without having to resort to diff type operations, and the guarantee that files will be made available at the remote site in the same order as they have been submitted for transfer locally. For example, if a file is transferred using the program shown in figure 4, processed using STATEL_EXEC, and then deleted using STATEL_UNLINK, the can rely on the fact that STATEL will process the correct file, even if an older file of the same name already existed.

Modularity STATEL has been designed to be very modular. There is a clear separation between the application, the STATEL API and the various Transfer Modules (TM). The STATEL routines linked to the application are responsible for the management of the VFS, and the generation of the PDUs. The TM is responsible for the transmission of the PDUs to the remote host, and the reception of PDUs from the remote host. As shown in Figure 4, the only information that must be exchanged between the STATEL Core and the TMs are these PDUs.


9

Figure 4: STATEL Modularity

This allows the applications linked with the STATEL Core and the Hub or Gateway to execute on different machines.

Security and reliability Security It is not trivial to provide classic password-type security with an API. First, the API doesn't necessarily know if there is a screen and keyboard available to the application, or, if they would be present, how to access them. Second, as STATEL can be used to support a number of applications, the password would have to be known by many people, which is not good security. So STATEL doesn't even try to use a password, but it goes a long way to render data transmission more secure by

1. Encrypting all transferred data using 3DES with a shared key agreed through the Diffie-Hellman algorithm,

2. Keeping the data it stores in encrypted format,

3. Requiring only non-specific access rights on the host machines,

4. Decoupling the user environment from the data transfer environment. The latter two points are very important in multi-user environments where the access to data belonging to a user will usually be restricted. Supposing the

Application

STATEL Core

STATEL API

TM API

Windows: DLL

Unix: program

Either on the sameor a different system

PDU

TM Net

TM FTP

TM HTTP

Gateway/Hub


10

transfer of the data happens asynchronously (cfr uucp), the telecommunications facility might not be able to access the data (or the data might have disappeared). STATEL solves this problem by making an encrypted copy of the original data (in the local VFS), and a corresponding PDU, which is made accessible to the data communications systems in an area accessible to them.

The encryption has the side benefit that data stored in the Windows version of STATEL is not accessible to prying eyes even if the STATEL directory resides on a FAT-type file system without access protection. No claims are made for the security of the encryption algorithm, but it should be good enough to deter a casual attack.

Reliability The PDUs are kept in the STATEL workspace until their reception has been acknowledged by the remote party (as an aside, these acknowledgments are part of the PDU, thereby eliminating the need for individual acknowledgments). If a PDU is lost in transit, the remote party requests a retransmission; this is possible because every PDU has a unique sequence number.

PDUs are processed in strict sequence number order, thereby ensuring that the data is presented on the remote host in the same order as it was generated on the local host. This is very important when the remote execution (STATEL_EXEC) feature of STATEL is used to trigger the processing of the transmitted information. STATEL guarantees that the data will be available before it launches the required process, making it impossible to process outdated or incomplete data sets.

Additionally, this feature enables the implementation of applications ensuring that transmitted data will be processed on the remote host, by following the transmission with an STATEL_EXEC call. Even if after this sequence, another data set is copied with the same STATEL file name, STATEL will have processed the first data set before replacing it with the second one.

STATEL can improve the reliability of the telecommunications facility by limiting the size of the PDU to a value comfortably supported by that facility. Files are automatically split over multiple PDUs, and become available only when the PDU containing the last part of the file is received and processed by the remote party.

New Features The latest release of STATEL has seen the introduction of some major new core features, such as:

• ZIP compression of the PDU payload


11

• Diffie-Hellman key agreement

• 3DES encryption of the PDU payload

• Automatic reset of a STATEL tuple with optional preservation of the VFS contents.

• Support of reporting from remote SNNs. Upon request of Eurostat remote SNNs send automatically and transparently extensive information about their configuration.

Of course, and honouring a long STATEL tradition, the new release is fully backwards compatible with existing releases. Upgrades can happen at any convenient moment, and production need not be interrupted.

The support for MS-Windows as a STATEL “server” host has also been improved, and the STATEL Hub now offers full gateway services.

I M P L E M E N T A T I O N

Chapter

2 Implementation This chapter describes the current STATEL implementation. The software is written in the 'C' language, and has been compiled on Solaris (from 2.6 to 9), Linux, AIX and HP-UX, as well as Microsoft Windows. The source code is compatible with ANSI 'C'.

First, the system and network topology of STATEL is discussed; one of the major attractions of the system is the ease with which it adapts not only to various protocols, but also to varying host environment and network topologies. Second, we detail the structure and features of the STATEL environment (messages and Virtual File System), indicating how it has been implemented on UNIX and Windows platforms. Third, the demands placed on the communications facilities are examined.

Topology STATEL has to be flexible to adapt to the many different variations presented by computer systems and networks. This goes from very simple one-computer setups, to LANs comprising different types of machines, with or without telecommunication servers such as the STATEL Hub, file servers, proxies, firewalls etc. The picture in Figure 5 shows how the STATEL system has been structured to enable it to adapt to a wide variety of host environment topologies.


14

Figure 5: STATEL Component Distribution

The API environment The STATEL API is linked with the application. The routines and subsidiary programs comprising the API are responsible for the maintenance of the local VFS, and the creation of the PDUs. These PDUs are stored in the Send Transit

Legend

Application

STATEL API

TM

TM

Gateway Hub

Computer system

Program or Library

Gateway Hub

Application

STATEL API + INCORP

TM

TM

Internet TESTA

Part of STATEL

Using shared file systems (such as NFS) allows the TM to be located on another system.

INCORP

The Incorporation process can be included in the API library.

A TM can be a separate process or part of STATEL libraries.

A TM can connect to the remote party through a Hub/Gateway or directly


15

directory. The PDUs received from the remote party are located in the Receive Transit directory. The API interprets the received PDUs through a subsidiary program, (INCORP in Figure 5), which effects the changes to the local VFS. The API has its configuration files, which contain all the necessary information to generate the PDUs and submit them to the Transfer Module(s), and the final address of the remote party if these need to be included in the PDU.

The Transfer Module Environment Its only mandatory link with the API environment is through the Receive Transit directory; the API looks for received PDUs in the Receive Transit directory, and keeps the generated (local) PDUs in the Send Transit directory until they are acknowledged by the remote party. From the latter directory they are submitted (copied) to the TM environment. This can be through a physical copy operation (the file containing the PDU is copied to the TM's input directory), or by passing the name of the file as a parameter to the TM program. In other words, it is possible to have a control transfer link between the two environments (ie the API launches the TM), but this is not required, as the PDUs can contain all the address information needed by the TM.

This architecture allows a very flexible configuration of the STATEL components. API and TM can be located on different machines, and have varying degrees of coupling. On a PC, the elements can be co-located, with all activities triggered by the execution of the applications. In larger, networked installations, a number of users can all transit through the TMs located on a telecommunications server. It is even possible to have a combination of these situations (e.g. tightly coupled for remote parties connected through the Internet, whilst using a central facility such as the STATEL Hub for parties connected through SMTP, or without access to the Internet).

One important advantage of this approach is that the Transfer Tools can be located in the De-Militarised Zone (DMZ).

The STATEL Gateway This is a Transfer Tool running on Unix-like systems and based on Open Source components such as the ProFTPD FTP server, the Apache HTTP server, and the isect Middleware Toolkit. Please note that the latest version of the STATEL Hub (described below) offers the same functionality as the Gateway, but in an MS-Windows environment.

The Gateway is basically an FTP server using a modified authentication mechanism. Instead of requiring a separate user name and password for each STATEL installation that want to use its services, the Gateway accepts the SNN and a computed password (a hash of the SNN). After a successful authentication, the only objects visible are two directories, one containing the PDUs destined for the connecting SNN, and one where PDUs for other installations can be deposited. The chroot() facility is used to ensure that the connection is limited to this specific area.

Basic principles


16

Recently an HTTP server has been built beside the FTP server, which uses the same principles for user authorization and, recuperation and uploading of PDUs described below.

The incoming directory on the Gateway contains all the PDUs (which can originate from several remote SNNs) destined for the connecting SNN. The only action required is to transfer all these PDUs to the incoming directory on the STATEL workstation, and to delete them after a successful transfer. The PDUs have a unique name designed to facilitate tracing and debugging, and which consists of the SNN and the PDU sequence number.

PDUs destined for other SNNs are simply uploaded in the outgoing directory. The FTP server notifies an external process (movepdu) as soon as the transfer is complete. This outside process examines the PDU, and moves it to the incoming directory of the destination SNN. It also changes the name to the SNN+sequence number convention. Incomplete or damaged PDUs are automatically removed.

A major advantage of the STATEL Gateway is that the configuration is totally automatic, and doesn’t require the creation of a user for each SNN that connects. The directories of both the sending and receiving SNNs are created if they don’t exist, and the data exchange process is operational after the first connection of both parties.

The Gateway requires very little management support, thanks to the automated creation of the incoming and outgoing directories. In addition, the fact that incompletely transferred PDUs are automatically discarded ensures that the disk space usage requires hardly any monitoring.

Contrary to the Hub (which started its career as a communications front-end for a number of STATEL installations), the Gateway does not support the STATEL protocol (retransmissions etc.). It is purely a “PDU switcher”, and it leaves all the protocol handling to the SNNs that connect to it.

The STATEL HUB The hub allows several STATEL workstations to combine and manage the traffic to and from the remote parties. There are several advantages to using the hub, as compared to having each workstation establish its own connections.

1. There is only one connection to the outside world to maintain, meaning lower maintenance costs and lower security risks, for example through the situation of the Hub in the DMZ.

2. Workstation communications occur at the speed of the local area network connection, not the connection to the remote party.

3. The transmission and reception of PDUs can be scheduled for those periods of maximum bandwidth availability.

Recuperating PDUs

Uploading PDUs

Automated configuration

Self-managing

No support for the STATEL protocol


17

The hub accepts connections from slave stations using the FTP protocol, modified to accept the STATEL-specific user name and password described above. When presented with such a combination, the hub allows access to the input and output directories specific to the station (identified by its SNN). If necessary, the directories are created.

At regular intervals, the hub contacts the STATEL gateway (which could also be a STATEL hub) and downloads the PDUs for all the SNNs (slave stations) it knows.

When the slave stations connect to the STATEL hub, they can download the PDUs obtained from the remote party, and they can upload the PDUs destined for the remote party, with the hub offering identical functionality as the FTP gateway. All the transfers to and from the Hub happen at LAN speeds, as opposed to the speed of the external line when the installations would be directly connecting to the remote gateway.

To optimize the use of the connection (Internet or private), the hub can be programmed to effect data transmissions at specific moments of the day, say after hours. When the lines are very busy, this usually results in smoother transfers than trying to push the data through a saturated connection, with the FTP timeouts this entails.

Even more important is that the hub proxies the STATEL protocol for the slave station. It handles requests for retransmission, and requests missing PDUs from the remote party, without intervention of the slave station. This vastly improves the reactivity of a STATEL installation compared with stand-alone stations who sometimes only run STATEL a few times per month.

The PDUs are inspected by the hub, and it replaces requests for retransmission it handles by a no-operation (NO-OP) component, allowing the slave station to process the PDU without effecting the retransmission a second time. Insofar as the hub fills the holes in the received PDU sequence before the slave station downloads the PDUs, retransmission requests to the remote party are also avoided.

The hub can use HTTP as a primary or backup protocol for the reception or transmission of PDUs. Exchange of PDUs using HTTP is useful for organizations that do not allow outgoing FTP connections.

Finally, the hub can, if required, use SMTP as a primary or backup protocol for the transmissions. SMTP support is vital to allow organisations that do not allow outgoing FTP or HTTP connections from their data centers.

Principles

Transmission scheduling

Protocol proxying

HTTP support

SMTP and POP3 support


18

Figure 6: The STATEL Hub

The streamlined access to the hub enables the configuration of a new SNN with a minimum of operator intervention at either end of the connection. When the hub receives a valid login sequence, it creates the limited file system hierarchy needed to service the connection, and updates its tables to reflect the new SNN. Similarly, the hub's equivalent at the other end creates its file system structures whe it receives the PDUs forwarded by the remote hub. This reduces the effort involved in creating a STATEL link to the creation of the appropriate SNNs at either end of the connection.

If the Hub is installed on a machine that's running 24/7, it will speed up retransmissions both to and from the remote party. Previous releases of STATEL for Windows were designed for desktop machines, that typically get switched off at the end of the day, and did not incorporate service-type functions to handle STATEL protocol issues without running STATEL enabled programs.

The Hub can be installed in the DMZ, ensuring that the STATEL installation on the LAN need not go beyond transferring data to and from the Hub. All the connections to the outside networks are then done by the Hub. Please note that the Hub never communicates with systems inside the LAN. All connections to the Hub are established by the STATEL installations.

System structure STATEL Sequence Numbers Whenever a PDU is generated, the local sequence number is incremented. All activities taking place within STATEL which are recorded in this PDU will

Diagram

Fast Configuration

Improved response times and reliability

Improved security

LAN

SNN-1

SNN-2

SNN-3

INTERNET

STATEL Hub


19

carry this sequence number as an indication of the relative time frame. These sequence numbers are transmitted to the remote party, where the last known sequence number is stored (this is the sequence number of the last incorporated remote PDU). In addition to the local sequence number, the PDUs carry the sequence number of the last remote PDU processed by STATEL. This serves as an (implicit) acknowledgment and allows the remote party to delete the acknowledged PDUs, which otherwise remain stored for a possible retransmission request.

STATEL messages When an application issues a STATEL_PUTMESSAGE primitive, the message body is copied to the PDU. No local trace of the message is kept. When the PDU is interpreted by the remote STATEL, the message body is stored in the MeSsage file (MS). The length of the message, and its position in the message file are stored in the Message Header file (MH). The STATEL_GETMESSAGE primitive can be used to retrieve the messages received from the remote party. Once they have been retrieved, messages disappear from the message queue; no acknowledgment of the message retrieval is sent to the remote party.

The STATEL Virtual File System (VFS) A distinct VFS is created for every tuple (local_SNN, remote_SNN); there is no communication between these file systems, and hence it is not possible to consider STATEL a network. At each side of the logical connection, the data files placed in the VFS by both sides are kept, together with the administrative information (meta-data) required to present the VFS and the data transfer process through the primitives.

File types The virtual file system contains the following types of files:

Type Comments D Directory files containing file or sub-directory names F Files, which are sent to / received from the remote party W Write-only file, the contents are sent to the remote party, no local copy is kept R Read-only file, the remote counterpart of a ‘W’ file (cannot be modified)

Table 2: STATEL VFS File types

File contents STATEL has been designed to consider files as closed entities, without any special features. The distinction between so-called text and binary files does not exist, and the structure, form, content and data representation are the sole responsibility of the applications using the VFS files. Thus, no attempts are made to convert line termination sequences, character sets, or to swap bytes in binary data. If an application expects its input to arrive in a particular format (say, ASCII lines separated by CR/LF sequences), the sending application has the chore to write the VFS file in that format, even if that is not the


20

representation used by its native OS. Alternatively, the receiving application could elect to transform the data received from its remote correspondents.

VFS Logical Structure The VFS offers a hierarchical directory organisation, with a number of sub-directories only limited by the available facilities. Figure 8 shows the logical structure of a typical VFS.

Figure 7: STATEL Virtual File System - logical structure

File attributes For each file in the VFS, the attributes shown in table 3 are maintained:

Attribute Type Size Comments File Type Character 1 F,D,W,R File name Character 64 File size Long integer -

Creator/Status Character 1 R,r,L Creation sequence Long integer -

Local Modification sequence Long integer - Remote Modification sequence Long integer -

Local Access sequence Long integer - Remote Access sequence Long integer -

Key Character 32 Encrypted Table 3: STATEL Virtual File System - File attributes

The Creator/Status field is used to indicate the origin of the VFS file, and the status of the transmission. Files which have not been completely received (because of PDU size restrictions) are marked r until the PDU containing the final part of the file has been received. Requests to open a partial file, or to obtain status information relating to the file, will fail. Version 3.4 allows access to partial files using the STATEL_FCNTL() primitive.

STATEL Root /

Application Directory B

Application Directory A

Data file

Data file Data file 1 Subdirectory Data file 2

Data file 1 Data file 2 Data file 3


21

The key is a security mechanism allowing a file or directory to be inaccessible unless the key is known. STATEL also keeps a system key in encrypted format in the SNN.ISF file. This system key can be used as an alternative to the file specific key.

VFS Internal Structure The VFS is implemented as a number of Operating System files, which are used to store the local copies of the data and system files. There is a VFS inode file (IN), which contains the attributes of the VFS files.

Figure 8: STATEL Virtual File System - Implementation

The VFS root directory is stored in another file (S0000000), and contains the file names and inode entries for each subsidiary file or directory. The OS file names storing the VFS data and directory files are generated from the inode number (ie. the file corresponding to inode #0 will have file name S0000000, etc.). As inodes are re-used when VFS files have been deleted, the number of entries in the OS directory is kept in check.

Locking incomplete files Since version 3.3, STATEL will not release files that have not been completely received. Typically, this situation occurs when a large file has been segmented and not all segments have been received. When a file is created through the incorporation of a PDU, the creator field is set to the lower-case "r". When the file is closed using STATEL_CLOSE(), the FMR.RL entry is written to the PDU. When this component is processed, the file's creator field is changed to upper case, and the file becomes accessible. Since version 3.4 STATEL

Name (64) inode

.

0

..

0

Statel_file 1

Welcome 2

STADIUM 3

Name (64)

inode

.

..

type size key

D 262

F 2233445

W 123456

D

sequences

C1

C7La0m0Ra7m7

C2La2m2Ra1m0

C7

File S0000001

L

creator

L

R

R

File S0000002


22

includes a mechanism to access such partial files through the STATEL_FCNTL() primitive. It uses the 8th bit of the field as an "unlock" marker. When the bit is set, access is granted even though the file has not been completely received.

Communication Principles Requirements The only functions required from the underlying transmission facility are the ability to send and receive files from the remote host. Before the user is granted access to the VFS through the use of the STATEL_INITIALIZE primitive, two activities take place:

1. As many PDUs as possible are obtained from the remote party.

2. These PDUS are integrated in the local VFS.

These activities can be triggered by the API, or can be considered to have occurred asynchronously. In other words, both the reception of PDUs from remote hosts, and the integration of these PDUs in the VFS, can be done by background tasks or other systems, provided the local STATEL host can access the appropriate directories. For instance, one could have a PC as STATEL host, with the STATEL environment located on a local volume, and accessing the telecommunications machine through a LAN. Another possible scenario is a UNIX machine hosting all elements of STATEL.

At the end of the STATEL session (STATEL_TERMINATE), the PDU which has been generated as a result of the activities in the VFS, is transmitted to the remote host, or placed in a directory where an asynchronous TM can transmit it to the required address. In the latter case, the PDU contains the address of the remote party, in a format acceptable to the TM (zero-lines or protocol-lines).

Protocol overview Whenever a modification to the data stored within STATEL takes place, a Protocol Data Unit (PDU) is generated. Every PDU has a unique sequence number (the CF file stores the sequence numbers between executions of STATEL based applications). The incorporation process (which can be run synchronously or asynchronously) processes the PDUs received from the remote party in strict sequence number order. When a gap in the sequence is detected, a retransmission request is added to the outgoing PDU. This request contains the number of the PDU last processed, and the number of the PDU causing the sequence break. When such a request is seen, the PDUs between those in the request are retransmitted.

Retransmission requests are processed out-of-band, i.e. they are executed even if the PDU which contains them cannot be processed because a sequence break has occurred.


23

Two mechanisms are provided to reduce useless traffic:

1. Requests for retransmission are only issued when a (configurable) time period has expired since the last request (set to 8 hours by default). This enables slow transmission facilities to process the request and subsequent retransmissions. This counter is also reset when (part of) the PDU gap is filled.

2. PDUs are not retransmitted every time the incorporation process is run, but only after a configurable time period has expired since the (group of) PDU(s) was last retransmitted.

In order to enable STATEL to retransmit missing PDUs, these are only deleted from the sendTransitDirectory when they are acknowledged by the remote party through the ISC.ID component (the remoteSequence item).

Tuning the PDU size Segmenting large files Certain transmission services impose limits on the size of the items they can transport. STATEL can adapt the size of the PDUs it generates to such a requirement through the use of the pduSizeLimit entry in the [Tuple Parameters] section of the tuple configuration file (CF). Whenever a PDU exceeds this limit, it is terminated and transmitted using the specified Transfer Module.

Delayed transmission The delayed Transmission parameter is used to group the transmission of all the PDUs at the end of the STATEL session (i.e. upon execution of the STATEL_TERMINATE() primitive). By default, each PDU is transmitted when it has been completed.

It is suggested to use delayed transmission when large volumes of data have to be transferred and the supporting communications facility is not very reliable. When this feature is used, STATEL will attempt to send the PDUs not yet transmitted whenever it is called, even though the original transmission might have failed, and will recover from machine shutdowns or crashes (if the file system is not damaged) during the transmission process.

Using delayed transmission will also ensure that PDUs do not grow to huge sizes when the communications facility is not available for prolonged periods of time. In its default configuration, STATEL appends components to the last PDU if the PDU wasn't transferred on the previous attempt, but the delayed transmission feature ensures that a new PDU is created to contain the additional information.


24

Encryption and compression Today’s security requirements mandate a better (or certainly a more accepted) encryption method than the simple systems used by STATEL 3. The latest version of STATEL supports a full-strength encryption mechanism based on OpenSSL, with Diffie-Hellman key agreement upon reset, and 3DES encrypted PDU payload. In addition, the data continues to be encrypted with STATEL’s original algorithm.

Key Agreement Principles To encrypt the data using the 3DES algorithm, a shared secret must be present in both installations. This shared secret should never be transferred through an unencrypted, unsafe medium, but on the other hand, it should also not be necessary to use another medium, least of all the intervention of humans. The Diffie-Hellman key agreement algorithm allows both end points of the STATEL connection to compute a shared secret based on values that can be transmitted over a clear channel.

When a STATEL tuple is reset using the STATEL_RESET() primitive, a PDU containing the ISC.RS element is sent to the remote party. This element contains the Diffie-Hellman ‘P’ parameter computed by the installation that initiated the reset, as well as the Public Key it computed based on ‘P’. When the remote installation processes the ISC.RS element, it uses ‘P’ to compute its own Public and Private keys, and then computes the shared secret based on the received Public Key. It then constructs a reply PDU containing its own Public Key in the ISC.KE element. Once the originating station receives the PDU with the ISC.KE element, it uses the received Public Key and its own Private and Public keys to compute in turn the shared secret.

Once both parties have computed the shared secret, the payload of the PDUs can be encrypted and decrypted by either of them.

Payload Encryption The PDUs cannot be fully encrypted because the information contained in the XFR and ISC elements would no longer be accessible, making it impossible for the Hub or the Gateway to function. Even worse, the incorporation function would no longer work, as it relies on the information in the ISC.ID element to determine which PDUs are destined for the various SNNs3. It is thus important to preserve the information in the envelope elements, while encrypting the information in the VFS, message and remote execution elements.

Rewriting the PDU If a shared secret is present, the STATEL PDU writer routine will encrypt all the non-header elements in the PDU after compressing them with the compression

3 Remember that the filename of the PDU is not used by STATEL, even though certain tools such as the STATEL Gateway use file names containing the SNN and sequence number.


25

algorithm from the zlib package. The compressed, encrypted data is then placed in an ISC.EN element and written back to the PDU. The ISC.TR element is then added as usual. Please note that the formal structure of the PDU remains the same as in previous versions of STATEL.

This approach ensures that STATEL components such as the Hub and the Gateway continue to function as usual, and that TMs can continue to add dynamic elements such as the LOG.MS elements added by the Hub to inform the remote party on the speed of the transfer.

Decrypting and expanding the PDU Once the PDU has been placed in the tuple’s incoming directory, the first access (using the pduaccess routines pduOpen() or pduOpenByName()) after the shared secret has been set will result in the ISC.EN element being decrypted and expanded. The resulting elements are written back to the PDU, and any trailing envelope or dynamic elements are added back with corrected line numbers. The resulting PDU is fully readable by the STATEL incorporation routines.

Even if the incorporation does not happen at that moment, the fact that the PDU is now decrypted does not dimish the confidentiality offered by the encryption:

• The PDU is located on the destination system,

• The data is still encrypted using the original STATEL algorithm.

The reader will remember that STATEL adds data to a PDU that was not transmitted (logically or physically). The fact that the decryption happens through the basic pduaccess routines ensures that a PDU that is still on the originating system will be decrypted before elements are added. When the PDU is closed and resubmitted to the TM, it is re-encrypted, with both the original and the additional elements contained in a single ISC.EN element.

Effect on non-transmitted PDUs

C O N F I G U R A T I O N P R I N C I P L E S

27

Chapter

3 Configuration Principles This chapter describes the layout of a STATEL installation, and describes the similarities and differences between the UNIX and MS-Windows versions. It also discusses the layout of the configuration files, general configuration principles, and the interaction with subsidiary programs.

Configuring the STATEL API Fundamentals STATEL uses host system directories and files to create its data stores, one for every tuple (local STATEL Nick Name/remote STATEL Nick Name pair). These stores are completely disjunct, and live in separate subdirectories. The PDUs destined for or obtained from the Transfer Modules (existing or furnished with STATEL) must be exported from or imported into this environment. To facilitate the communication with (protected) transfer systems on multi-user systems such as UNIX, the export and import functions are executed by separate programs, which can be given the required privileges. By creating separate instances of these programs, multiple transfer systems can be used in a safe and transparent fashion. When the export and import programs are the transfer system (i.e. when using ncftpget and ncftpput on a UNIX system), they can be used as direct replacements for the Transfer Modules. This flexibility requires that prior to installing and configuring STATEL, care should be taken to analyse the topology of the host system, and to prepare a layout of the installation.

In certain circumstances, a STATEL system might involve a number of separate OS user IDs before the PDUs leave the originating systems or network. This is illustrated in Figure 9.

Chapter

3


28

Figure 9: Layout of a STATEL Installation

The programs TM and INCORP are used to export PDUs to and import PDUs from the Transfer Module environment. The latter program makes the link with the data transmission facility (FTP or SMTP). The application and STATEL itself (by virtue of it being a library) belong to two different user IDs. On UNIX machines, the access problems are overcome by using the SETGID (Set Group ID) feature, which allows programs to assume the privileges of the group to

receive

Legend

Application

STATEL API

TM

Gateway Hub

Computer system

Program or Library

Gateway Hub

Application

STATEL API + INCORP

TM

Internet TESTA

Part of STATEL

INCORP

VFS

send

config

config

UID-Application GID-STATEL

UID-STATEL GID-STATEL

UID-Gateway GID-Gateway

UID-TM GID-STATEL


29

which they belong, even if the user who launches them belongs to another group. Figure 9 emphasises how important careful planning and preparation is for a successful installation.

Defining the STATEL universe To enable STATEL to locate its main configuration file, the environment variable STATELBASEDIR should be set to point to the directory containing this configuration file. When using the UNIX operating system, commands comparable to the following lines can be used (to make the situation permanent, they should be added to the start-up files for the Bourne or C-shell).

$ STATELBASEDIR=/home/statel ; export STATELBASEDIR (using the Bourne or Korn shells) sibelius:/usr> setenv STATELBASEDIR /home/statel (using the C-shell)

As all STATEL activity is determined by this variable, it is quite possible for the same user or the same application to have access to many different STATEL universes.

STATEL for Windows also uses the STATELBASEDIR environment variable, but alternatively will read the base directory specification from the STATEL32.INI file (from section [STATEL]):

[STATEL] STATELBASEDIR=C:\STATEL3 ...

The STATEL32.INI file is located in the Windows directory of the system (x:\Windows or x:\Winnt, with x representing the installation drive). Other items that can be specified in the STATEL32.INI file are:

Variable Description STATELBASEDIR Location of the STATEL subdirectory LANG Language used for the error messages STDEBUG Debug level STDEBUGFILE File the debug output will be appended to Table 4: STATEL32.INI parameters

Configuration file expansion rules Whenever a file name is specified in a STATEL configuration file, the following expansions are performed (see Table 5).

Specification Expansion First pass $VAR/name VAR is interpreted as an environment variable. @PAR@/name PAR is a parameter defined earlier (CF only) Second pass ~/name name in the tuple's directory name name in the STATEL Base directory /name name without change


30

Table 5: STATEL File name Expansion rules

Through this feature, the installation can be made independent of the layout of the local file systems.

In addition, environment variables and parameters can be used in other configuration file entries to make the configuration process more coherent and less error-prone, or to enable system-wide parameters to be changed without editing all the configuration files.

Configuring the STATEL Nick Names STATEL's main configuration file is called SNN.ISF, and is located in STATELBASEDIR. It is an editable file with the following layout:

[Statel Nick Names] # Local STATEL Nick Name (SNN, max. 35 chars, cfr. EDIFACT) localSNN=Eurostat # Remote STATEL Nick Names, Tuple directory remoteSNN=CSO-Germany,CSOGER remoteSNN=CSO-GB-London,CSOGBL remoteSNN=NIS-Belgium,NISBEL remoteSNN=CBS-Heerlen,NISNLH [System Parameters] # Directory where BAD PDUs are moved to badPDUDirectory=/BAD # Default send transit directory (~/ = tuple directory) sendTransitDirectory=~/send # Default receive transit directory receiveTransitDirectory=~/receive # Interval between retransmit requests (system default: 8hrs) # or between retransmissions of the same PDUs resendFrequency=1440 [Registered Users] User = statel User = stadium2

All the STATEL nick names (local and remote) are grouped in the section [Statel Nick Names]. There is one localSNN entry, and there can be as many remoteSNN entries as necessary. The second item in the remoteSNN entry is the base directory for the files comprising the VFS for that remote party. This entry is mandatory, and it allows the selection of SNNs not limited by the host operating system directory naming limitations (in the current implementation the SNN can be up to 35 characters long). Please note that these tuple directories are always sub-directories of STATELBASEDIR. As the file is editable, make sure that there are no duplicate entries. Assigning the same tuple directory to two different remoteSNNs would lead to very confusing results. The program addtuple checks that no duplicates exist and that existing tuple directories are avoided.

The [Registered Users] section is used to limit the access to STATEL to specific users. For obvious reasons, it is only implemented in the UNIX version of STATEL.


31

Setting the system defaults The parameters set in this section can be overridden by entries of the same name in the tuple configuration files.

Transit Directories For each active tuple, STATEL maintains two transit directories. The send transit directory contains the non-acknowledged PDUs generated by the local STATEL party. As the PDUs for the various tuples have similar names, the send transit directories need to be unique for each tuple. The receive transit directory is the directory where STATEL will look for PDUs received from the remote party. The [System Parameters] section is used to define system-wide defaults for the PDU transit directories. These settings can be overridden using the tuple configuration file. The tilde (~) is used as a shorthand for the active tuple directory and should not be confused with the C-shell convention.

Retransmit delay To avoid issuing too many retransmission requests, STATEL applies a default time interval between retransmission requests of 8 hours (28800 seconds). This default can be changed system-wide by adding the retransmitDelay parameter, indicating the delay (in seconds) between successive retransmission requests.

Remotely executable commands The commands which can be executed by all remote parties are defined in the file CO located in STATELBASEDIR. The format of the file is as follows:

[Commands] statelName=localName statelName=localName ...

The CO files located in the tuple directories can be used to override the equivalences defined for all correspondents, or to make certain commands available to selected parties only.

Logging options Please refer to Appendix A for a thorough discussion of the logging features (both local and remote) offered by STATEL.

Configuring the tuples The tuple directory defined in the main configuration file contains all the files relative to the STATEL activities of the tuple. The only file which can be edited by the user is the tuple configuration file CF. The file has a similar structure to the SNN.ISF file.

# Generated by addtuple.c 1.8 on Wed Apr 28 16:50:53 1993 [Transfer Agents]


32

transmitTransferAgent=SYNC,/usr/bin/uuto,@PDU@,@Address@ receiveTransferAgent=ASYNC incorporationAgent=SYNC,$HOME/bin/incorp,@remoteSNN@ [Tuple Parameters] receiveTransitDirectory=/usr/spool/uucppublic/receive/statel1/joost Address=joost!statel2 parametersInEnvironment=0 Address=C=BE;A=RTT;P=CEC;O=EUROSTAT;S=STATEL [Sequence Numbers] # Set both numbers to zero to re-initialise STATEL localSequence= 2 remoteSequence= 2

Defining the Transfer Agents The Transfer Agents are programs which can be executed to communicate with the TM, or they can be the TM itself. There are a number of entries in the [Transfer Agents] section, that determine how PDUs are transmitted, received and interpreted (the latter item only for UNIX).

The keywords used to define the receive and transmit actions are different for the Windows version (originally to enable the same STATEL installation to be used with both the MS-DOS and MS-Windows versions of STATEL). The Unix version uses the same keywords as the MS-DOS implementation, and doesn’t recognise the Windows keywords.

Name Platform Description ReceiveTransferAgent Unix Program to execute to receive PDUs from the

remote party. TransmitTransferAgent Unix Program to execute to transmit a local PDU to

the remote party. WindowsReceiveTA MS-Windows Name of the .DLL containing a function with

the same name, used to receive PDUs from the remote party.

WindowsTransmitTA MS-Windows Name of the .DLL containing a function with the same name, used to transmit one PDU to the remote party.

WindowsMultiTransmitTA MS-Windows Name of the .DLL containing a function with the same name, used to transmit a number of PDUs to the remote party

Table 6: TM Entry point specifications

The transmitTransferAgent (or WindowsTransmitTA function) is invoked by STATEL_TERMINATE to submit the local PDU to the transfer facility for transmission to the remote party. Depending on the support for the protocol, the program launched here can be either a submission program, or the TM itself. In the example above, the protocol used is uucp, the TM is uucico (an existing part of uucp), and the program used to submit the PDU to the TM is uuto. The first item of the entry can be either SYNC or ASYNC, indicating that the program is launched by STATEL (SYNC) or executes independently (ASYNC). The second item is the program name, the remaining entries are the parameters passed to the program. These parameters can be hard-coded constants, environment variables or macros, which are expanded by STATEL at


33

run time. Table 6 gives the macros which can be used to build the transfer agent command lines.

Macro Name Description

@PDU@ The path name of the current PDU @BASEPDU@ The terminal file name of the current PDU @localSNN@ The name of the local party @remoteSNN@ The name of the remote party @Address@ Value of the Address= entry in the [Tuple Parameters] section. @loginBase@ Base directory on the remote system, used when STATEL

should contact a server (eg Kermit) on a remote host. This entry defines where on the remote host the PDUs can be retrieved and stored.

@loginInput@ Subdirectory in loginBase where the PDUs generated by the remote party are kept for retrieval. Once retrieved they can be deleted by the party calling the server.

@loginOutput@ Subdirectory of loginBasewhere the local PDUs should be stored so that the remote party can access them.

@receiveTransitDirectory@ Where INCORP looks for new remote PDUs @sendTransitDirectory@ Where the API generates the PDUs, distinct from the loginInput

directory. Table 7: Tuple Configuration Macros

The receiveTransferAgent is invoked by STATEL_INITIALIZE to obtain as many PDUs as possible from the remote party. In our example, nothing is done, as uucp delivers all the PDUs it receives in a known, accessible directory on the local host. The first entry of the item is ASYNC, and there are no further items.

The incorporationAgent entry defines the program invoked by STATEL immediately after the execution of the receiveTransferAgent. The standard program delivered with STATEL is called incorp. This program takes the remote SNN as its unique argument, and inspects the configuration file to obtain the name of the receiveTransitDirectory. This program does not rely on file names, but checks all the files in the directory to see if they are PDUs for the tuple under consideration. Files that are not for the tuple, or are formally incorrect, or incomplete, are ignored. It is thus possible to process PDUs when a new PDU is being received (copied) into the directory, and to receive PDUs for many tuples in the same directory when the TM supports unique naming of received files.

Setting Tuple Parameters The [Tuple Parameters] section can be used to override the defaults established by the [System Parameters] section of the master configuration file. In the example in figure 12 this feature is used to instruct STATEL to use the uucp reception directory rather than the system default as receiveTransitDirectory.

The original encryption supported by STATEL was primarily meant to protect the data stored in the VFS from being used without going through the API. It was also useful as a basic protection of the data in the PDUs.

Overriding defaults

Configuring Encryption


34

Entering Protocol Lines The [Protocol Lines] section is optional, and is used to instruct the TM when it cannot be passed the parameters by the API routines. This would normally be the case when the TM executes as a separate process, such as the X400 TM When present, the entries in this section are copied into the PDU as XFR.IN components in the order they appear in this section. Please note that each entry must be preceded by the keyword line= or they will not be copied into the PDU.

Configuring the Transfer Modules As the Transfer Modules consist of existing programs, such as ftp, which are used directly or through a wrapper, and custom written TMs (e.g. ftp under MS-Windows), there are no uniform rules for their configuration. It is advisable to call on people familiar with the protocols and programs to help with their configuration and installation. Manuals on these programs are, as far as possible, furnished with the STATEL package, or are part of the protocol support software.

The STATEL error reporting mechanism can be used to offer the user multi-lingual messages for errors reported by the transfer modules. For each TM used by STATEL, a range of error codes can be entered in the error message file (ERRCODE.lang); the range used by each TM should be configured in the [Tuple Parameter] section of the CF files.

Keyword Effect receiveErrorBase Defines the start of the error message range for the receive Transfer Module used transmitErrorBase Defines the start of the error message range for the transmit Transfer Module used incorporationErrorBase Defines the start of the error message range for the incorporation Module used Table 8: Error message range definition

Please note that error messages below 255 are reserved for the STATEL API.

ftp Description ftp is the program implementing the client side of the Internet FTP protocol. All recent machines come with one or more implementations, but not many comes standard with a batch-capable FTP client. MS-Windows comes with a very primitive command line FTP client, which is totally unsuitable for use with STATEL.

Support strategy The ftp program has been designed mainly for interactive use, and requires the user to have a login facility on the remote host. UNIX implementations often insist on using the terminal to query the user for this password, making batch use of the program rather difficult. Part of the STATEL package is tmftp, which allows a flexible background use of the ftp program. It is also possible to use a third-party FTP program such as ncftp.


35

The MS-Windows version of STATEL contains a non-interactive implementation of the FTP protocol over Winsock.

The STATEL (FTP) Gateway is normally used to act as an intermediary between external and internal STATEL installations. The Gateway is installed in the DMZ, and both external and internal installations poll it to obtain their PDUs from their incoming directory. To send PDUs to a remote party, an installation copies it to its outgoing directory, and the Gateway instantly moves it to the appropriate destination directory. To avoid having to configure a separate user for each external installation, the Gateway uses a modified version of the FTP login procedure. The user name (login) used is the SNN, and the password is the HAVAL hash of the SNN, represented as 32 hexadecimal characters. Once connected, the FTP client sees only and incoming directory (where all the PDUs it needs to collect are stored), and an outgoing directory, where it can copy a PDU for a remote party.

HTTP Description The HTTP method of transferring files covers situations, in which for various reasons a remote client does not have ftp access to the STATEL gateway. HTTP support under Windows is provided for both standalone installation and hub.

Support strategy HTTP is used to connect a remote installation (or the STATEL hub) with the gateway. It is used neither to connect the internal installation (alpha) with the gateway, nor a standalone installation of a LAN with the Hub. For these cases batch-capable ftp clients are used (tmftp, ncftp, wtmftp). Remote installations use GET method to receive PDUs from the gateway and POST method to transmit them. The PDU’s content is transmitted as raw binary data wrapped with a proprietarily encoded envelop, which carries the necessary control information (sending SNN, PDU filename, statistics of the transfer). HTTP server shares the same directories with the FTP server and applies the same principles described above for user authorization and data storage. SMTP e-mail systems Description SMTP support under Windows is provided through the STATEL Hub, as it is the only STATEL component that runs as a service. Until now, STATEL for Windows has taken the approach that the availability of the system would follow workstation, not server patterns, as opposed to the design of STATEL for Unix. Here, SMTP support is provided through the native email interface, in keeping with the STATEL design philosophy of a maximum re-use of standard components.


36

Support strategy The STATEL PDUs are encapsulated in a MIME-compliant email message, containing a body part with a message readable by the human that might mistakenly receive a STATEL PDU. The PDU itself is encoded using base-64 encoding.

The STATEL Mail Robot can handle situations where the emitting MTA adds another encoding to the PDU, or changes the base-64 encoding to older standards such as uuencode.

The destination email address should be made available through an XFR.IN protocol line, as shown below:

# Generated by $RCSfile: addtuple.c,v $(Release $Revision: 1.6 $) # on Wed Nov 13 13:25:20 2002 [Sequence Numbers] localSequence=22 remoteSequence=11 lastResendTime=0 # Derived from template file internetmail.tem [Transfer Agents] receiveTransferAgent=ASYNC transmitTransferAgent=SYNC,$STATELBIN/udcp,@PDU@,/home/statel/SMLR/Data/OutgoingReady/pdu incorporationAgent=ASYNC,$STATELBIN/incorp,@remoteSNN@ [Tuple Parameters] receiveTransitDirectory=/home/statel/SMLR/Data/IncomingReady/pdu sendTransitDirectory=~/send Address=@remoteSNN@ pduSizeLimit=500000 useFileLocking=YES remoteProtocolVersion=4.1.4

[Protocol Lines] line=Address=swc(a)buht.cbs.nl line=ID=@remoteSNN@


37

Configuration at Eurostat The following figure shows the actual STATEL configuration at Eurostat. Please note that STATEL is also used to transfer data to and form the secure environment through a separate gateway installed on pepusch (a very old Sun SPARCStation 10).

Figure 10: The STATEL installation at Eurostat

Data from the Member States arrives either through the STATEL Gateway installed on yam or through the STATEL Mail Robot, which supervises a POP3 mailbox. The PDUs arriving in the Gateway through FTP or HTTP are “routed” to the destination SNN by the middleware system Isect and the collaborating program movepdu (a so called “isect worker”). The programs that compose the Isect system (isectd, isdexecd, …) are created by the same building procedure as the internal STATEL installation.

The messages arriving in the mailbox are split in their body parts, and each of these is examined. STATEL PDUs are copied to an output directory (typically the “receive” directory of the corresponding tuple), or presented to STADIUM-4 through the special STATEL tuple (STADIUM,eurostat-1).

The Mail Robot is also used to send PDUs to the tuples using SMTP, or standard mail messages to remote parties. To send an email message, STADIUM copies the file to send to a STATEL output directory which, by its name, identifies the recipient. The correspondence between recipients and destination addresses is defined in a special configuration file. The Robot uses this address file to send the data files as MIME attachments to the appropriate address, using the system’s “sendmail” command. During 2005, STATEL Mail Robot was renewed. A new set of modules with improved configuration and data processing mechanism replaced the old apparat (tmuni). The new system is also capable to recognise and treat eDAMIS e-mail messages.

alpha (eurostat-1)

[UNIX]

yam (gateway)

[UNIX]

pepusch (gateway)

[UNIX}

SMLR (Mail Robot)

[on alpha]

receive

send

verdi (distribution)

[UNIX]

alida (secure-1)

[UNIX]

seppuku (secure-2)

[UNIX]

FTP STATEL

HTTP

POP3

SMTP

D E S I G N I N G S T A T E L A P P L I C A T I O N S

39

Designing STATEL applications This chapter deals with the development of STATEL based applications. We first discuss the available language bindings, which allow STATEL to be used through 'C', C++, Perl and Tcl/Tk. The next section describes the UNIX command line utilities. Finally, we also describe the development of a minimal data capture program using STATEL.

Language bindings The native implementation of STATEL can be used directly in 'C' and C++; in addition, language bindings for Perl and Tcl/Tk have been developed.

STATEL for Perl A Perl XS module has been written to allow the use of STATEL related statements in Perl programs. The STATEL module has to be loaded through the use STATEL; instruction.

BEGIN { $| = 1; } END {print "Abort: Compilation error or cannot load modules\n" unless $loaded;} use STATEL; use Net::FTP; use Net::SMTP; use Convert::EBCDIC qw(ascii2ebcdic ebcdic2ascii); $loaded = 1; # Get the local SNN $olaSTATEL_snn = STATEL::localsnn(); $base_dir = STATEL::basedir();

Overview of Perl/STATEL commands The STATEL module has to be loaded through the use STATEL; instruction. It adds the following commands to Perl:

Command Type Description

STATEL::localsnn() String return the local SNN STATEL::basedir() String return the STATEL base directory STATEL::snnlist() Array return the remote SNNs STATEL::init($remote_snn) Boolean Initialize STATEL STATEL::terminate() Void Terminate STATEL session

Chapter

4


40

Command Type Description STATEL::open($filename,$mode,$key) Int Open a STATEL file STATEL::close($handle) Int Close an open STATEL file STATEL::read($handle,$buffer,$size) Long Read data from a STATEL file STATEL::write($handle,$buffer,$size) Long Write data to a STATEL file STATEL::puts($handle,$string) Long Put a string to a STATEL file STATEL::gets($handle) String Get a string from a STATEL file STATEL::copyfile($direc,$src,$dest,$key) Boolean Copy a file to/from STATEL STATEL::seek($handle,$offset,$whence) Long Manipulate the file pointer STATEL::tell($handle) Long Return the current file pointer STATEL::unlink($name,$key) Boolean Unlink (delete) a STATEL file STATEL::mkdir($name,$key) Boolean Create a STATEL directory STATEL::rmdir($name,$key) Boolean Remove a STATEL directory STATEL::chdir($name,$key) Boolean Change to a new working directory STATEL::getcwd() String Return the current working directory STATEL::fcntl($name,$key,$action) Boolean Manipulate file attributes STATEL::stat($name,$key,$result) Boolean Fill $result with status information STATEL::fstat($handle,$result) Boolean Fill $result with status information STATEL::istat($inode,$key,$result) Boolean Fill $result with status information STATEL::error() String Return the error message STATEL::errorcode() Int Return the last error code STATEL::opendir($name,$key) Int Open a directory and return a directory

handle STATEL::closedir($handle) Void Close an open directory

String Return the name of the next file in the directory

STATEL::readdir($handle)

Array Return all the file names in an array STATEL::unlock($snn) Boolean Unlock a locked SNN STATEL::putmesg($type,$mesg) Boolean Send a STATEL message

String Read the next STATEL message STATEL::getmesg($type) Array Return all matching messages in an

array Table 9: Overview of Perl/STATEL commands

STATEL for Tcl/Tk Designing graphical STATEL applications under UNIX becomes very easy with Tcl/Tk. The STATEL module integrates closely with the Tcl I/O mechanism, and once a STATEL file has been opened, it can be accessed with the same instructions as native OS files.

Overview of Tcl/STATEL commands In order to access the STATEL API through Tcl, the STATEL module should be loaded:

lappend auto_path $env(STATELBIN) lappend auto_path /home/statel/lib lappend auto_path . package require tclStatel

The following table gives an overview of the STATEL commands available through the tclStatel loadable module:

Command Parameters Description init $snn Start a STATEL session


41

exit Terminate a STATEL session (synonym: term)

term Terminate s STATEL session (synonym: exit)

open $file -mode 'R|W|O|A' -key $key Open a STATEL file stat file | channel | inode -key $key Get STATEL status information dir $dirname Return directory contents rm -key $key $filename Remove a STATEL file (synonym: delete) delete -key $key $filename Remove a STATEL file (synonym: rm) cd -key $key $dirname Change the STATEL working directory pwd Return the STATEL working directory fcntl $filename P | C Change file parameters mkdir -key $key $dirname Create a STATEL directory rmdir -key $key $dirname Remove a STATEL directory rename -key $key $oldfilename

$newfilename Rename a STATEL file

local Return the local STATEL Nick Name remote Return a list of the remote SNNs error Return the last error message errcode Return the last error code version Return the STATEL version unlock $snn Unlock a locked SNN islocked $snn Return TRUE if SNN is locked Table 10: Overview of Tcl/Tk STATEL commands

Programming example The following code fragment shows the use of the STATEL commands to display a tree representation of the remote SNNs.

# Load the SNN tree view proc LoadSNNPanel { } { global param lockedSNN # Determine which SNNs to display if {[string compare $param(restrictSNN) "*"] == 0} { set SNNList [statel remote] } else { set SNNList $param(restrictSNN) } # Load the lines in the snn list foreach snn $SNNList { if { $param(expandSNN) == 1 } { # Open the STATEL VFS if {[catch {set stat [statel init $snn]} errmsg] != 0} { Tree:NewItem .w /$snn -image ilock incr lockedSNN } else { # Display the contents in the tree Tree:NewItem .w /$snn -image isnn TreeUpDirectory .w $snn / 0 0 catch {statel exit} catch {unset SStatus} } } else { if {[statel islocked $snn] == "1"} { Tree:NewItem .w /$snn -image ilock incr lockedSNN } else { Tree:NewItem .w /$snn -image isnn } } }


42

}

The Tcl/Tk STATEL commands do not have special commands to write to a STATEL file, or read from a STATEL file. Once a file has been opened with the statel open command, a STATEL file is accessed with the normal Tcl file access commands:

1 if {[catch {set type [statel stat file \ 2 $statelFileName -item type]} errmsg] || \ 3 $type != "D" } { 4 if {[catch {set dst [statel open \ 5 $statelFileName -mode $mode]} errmsg]} { 6 ecc_dialog .warning "$errmsg" \ 7 "Cannot create file $statelFileName" \ 8 "questhead" 0 Dismiss 9 return 10 } 11 12 set src [open $fileName] 13 fconfigure $src -translation binary 14 fcopy $src $dst 15 close $src 16 close $dst 17 set size [statel stat file $statelFileName -item size] 18 set cseq [statel stat file $statelFileName -item creationSeq] 19 set raseq [statel stat file $statelFileName -item remoteAccessSeq] 20 set cdate [statel stat file $statelFileName -item ctime] 21 set creator L 22 } else { 23 set dst [statel open $Dialog_prompt(entry) -mode W] 24 close $dst 25 } 26 if { $SStatus(lastAck) < $cseq } { 27 set comstat "(Pending)" 28 } elseif { $raseq != 0 } { 29 set comstat "(Accessed)" 30 } else { 31 set comstat "(Received)" 32 }

Lines 14-16 show how a local file is copied to STATEL through the standard Tcl file access commands. The STATEL file is closed through the same command as the local file.

STATEL Command Line Utilities The STATEL primitives have been used to implement a number of command line utilities. These are both examples of the use of the STATEL primitives, and production quality file transfer and VFS management utilities that can be used in shell scripts or from the command line.

VFS utilities The utilities are patterned after the equivalent UNIX and MS-DOS programs, and allow the user to create and remove VFS directories and files, and copy local files to the VFS. They are not efficient when an application needs to transmit several files (or perform advanced file management), as each execution results in one PDU. The name of the remote party is always followed by a '+' sign, inspired by the convention used with NFS, where the machine name is followed by a colon (':').


43

vfscopy, vfsmove and vfscat This command copies files to and from the VFS. When copying files to the VFS, the command behaves very much like its UNIX equivalent: there can be many source file names, and they are all copied to the VFS directory with their original base names. This means that the last part of the file name (ie. from the last directory separator onwards) will be used as the VFS name. Conversely, the utility can be used to copy VFS files to the local file system. If called with the name vfsmove, the program will move the file to and from the VFS (i.e. the original file will be deleted after a successful copy. Calling the program via the vfscat name will copy a VFS file to the standard output. In this case, the program takes only a single parameter, which should reference the VFS.

To copy file JULYDATA.TXT in directory C:\MAINFR to subdirectory IDEP of remote party de-stba-1, the following command can be used:

K:\TEST> C: C:\> cd MAINFR C:\MAINFR> vfscopy JULYDATA.TXT de-stba-1+/IDEP C:\MAINFR>

The file name on the host (JULYDATA.TXT) is used in the VFS. To change the name of the file in the VFS from the host file name, use the command as shown below:

K:\TEST> vfscopy C:\MAINFR\JULYDATA.TXT de-stba-1+/IDEP/july_data_file K:\TEST>

vfsdir This command lists the VFS directory for a tuple. It can also display the number and types of the messages (-m option).

The following command lists the VFS directory for remote party de-stba-1:

K:\TEST> vfsdir/al de-stba-1+IDEP[CR] . D L 216 [47] .. D L 504 [0] JULYDATA.TXT F L 123456 [70,70<0>,70<0>] july_data_file F L 123456 [71,71<0>,71<0>] K:\TEST>

vfsrm This command removes files from the VFS.

To remove file july_data_file from the subdirectory IDEP from the VFS of remote party de-stba-1, one could issue the following command:

K:\TEST> vfsrm de-stba-1+/IDEP/july_data_file K:\TEST>

Example

Example

Example


44

Please note that these utilities require the full VFS path name to be given. Alternatively, the STATELCDW environment can be set to the base STATEL directory.

vfsmkdir This command creates VFS sub-directories using the STATEL_MKDIR primitive.

The following command should be used to create the IDEP subdirectory.

K:\TEST> vfsmkdir de-stba-1+/IDEP K:\TEST>

vfsrmdir This command removes empty VFS sub-directories, by passing the subdirectory name, and the optional key, to the STATEL_RMDIR primitive.

vfsmesg This command sends and displays STATEL messages. It was written to exercise the STATEL_GETMESSAGE and STATEL_PUTMESSAGE primitives.

vfsulock This utility program is used to remove lock files from a tuple directory, and, used with the -v option, to display the status of a tuple.

statel:athena> vfsulock -v gr-eudyn-1 Local SNN : euro-athena4 Remote SNN : gr-eudyn-1 Protocol : 4.1.4 PDU generation : 1 Encryption : Yes Current Local Sequence : 4 Last PDU acknowledged : 2 Remote Sequence : 5 Statel Base Directory : /home/statel/DATA Tuple Directory : gr-eudyn-1 Send Transit Directory : /home/statel/DATA/gr-eudyn-1/send Receive Transit Directory : /home/statel/Auto Resend Frequency : 28800 seconds Last PDU(s) retransmitted at : **Fri Feb 3 15:05:25 2006 List of retransmitted PDUs : (3) Last resend request at : none active Last transmission status : OK statel:athena>

vfsrep This is a shell script – wrapper of vfsulock, which adds by default the –v option.

vfspart This utility allows partial files to be listed witrh vfsdir, copied, and removed. It can both lock and unlock a partial file, so a file can be inspected and returned to a locked state pending the arrival of the PDU(s) containing the missing parts of the file.

Example


45

vfspass This utility is used to set or change the system key. This key can be used as an alternative for the optional key associated with a STATEL file or directory.

vfspredir This utility can be used to list the files contained in PDUs that have not yet been processed (due to one or more missing PDUs). It scans the PDUs in the receiveTransitDirectory, and shows only those files that are completely contained within the available PDUs. In other words, if it cannot find a FMR.RL component for the file, the file will not be shown, and it won't be possible to extract the file with the vfsextract command.

vfsextract The vfsextract command is used to extract files from PDUs that have not yet been processed. The file appears in the VFS in exactly the same fashion as when the PDUs were incorporated normally, and will not be re-extracted when the PDUs are incorporated after the reception of the missing PDUS. Please note that this utility should be used by knowledgeable staff only, as it could cause data loss or the processing of old data.

vfsreset The vfsreset command is used to reset the tuple and re-negotiate the shared secret. The latest version supports four (4) types of reset:

1. Reset with re-negotiation of a shared secret

2. Reset with re-negotiation of a shared secret and preservation of the VFS 3. Reset without re-negotiation of the shared secret (only VFS deletion) 4. Reset with neither re-negotiation nor VFS deletion (only nullification of

sequences)

vfsrun This command initiates the execution of a STATEL command, configured to run on the remote host.

vfscheck This vfscheck command check for consistency the i-node files of a VFS and reports information for each i-node.

PDU incorporation programs The utilities of this category receive and process PDUs.

incorp This program incorporates STATEL PDUs. It examines the receive transit directory for the given SNN and if it finds any incoming PDUs, it processes


46

them. PDUs for different STATEL tuples can be located in the same directory, and the file name is not significant.

stincd This is the STATEL incorporation daemon - designed to minimize the overhead of incorporating STATEL PDUs in large STATEL installations. Once the daemon has been started, it monitors the receive transit directories defined for the STATEL installation. As long as the directories remain unmodified, stincd uses minimal resources. When a new PDU arrives, stincd scans the directory once and is building a list of PDUs for each SNN.

ftppoll This is a shell script which runs periodically and fetches PDUs from the DMZ to the internal installation.

Administration utilities and external tools The utilities of this category are used by the system administrator to perform various administrative tasks or to create accessorial scripts.

addtuple This command creates a new STATEL tuple (combination of local STATEL nick name and remote STATEL nick name).

vfsadmin The command vfsadmin is used to request from a remote SNN the sending of system reports.

vfsdiag, vfstat, vfsurg They are shell scripts, used to observe the diagnostic, statistics and urgent log file respectively.

spr This program is called from the scripts above to prints the log entries in a format convenient for humans.

pduaccess This program scans the receiving or sending transit directories for a specific SNN and prints the list of PDUs found.

bulog This program is used for the rotation of log files. As arguments, it takes the maximum allowed file size and the maximum number of rotated files that have to be kept.

slog The program slog takes as argument a variable number of words and concatenates them using colons. The created string is prefixed with the current


47

time in seconds and printed in the standard output. It be called from shell scripts to log various events in the STATEL log files

Haval This program takes as argument an SNN and prints in the standard output its haval hash. Useful for configuration of batch ftp clients.

tmftp This program is a batch FTP client used as transfer module.

tmhttp This program runs in the STATEL Gateway as transfer module for reception of PDUs via HTTP.

ncftpget/ncftpput The third party programs ncftpget and ncftpput are batch ftp clients (more information in http://www.ncftp.org) which are used by STATEL as transfer modules.

Implementation The source code for the applications described in the previous chapter is delivered with the STATEL development system in machine readable format. Please feel free to incorporate elements of these programs in your own developments.

Developing with STATEL In this section, we will step through a complete program. This should give you a reasonable feel for STATEL, both its features and its limitations.

General concepts STATEL goes far beyond the traditional approach to file transfer, which requires you to prepare the data to be transferred off-line (at least as far as the transmission is concerned), in a local file. The application used to prepare the file is usually completely unaware of the state of the transmission process, and whether the remote party has (or hasn't) used the previous data set.

STATEL enables the developer to check the status of the transmitted data in a completely natural way, i.e. through comparing file status information obtained at different moments.

Monitoring the data exchange process Using the information returned by the STATEL primitives, the application can monitor the progress of the data exchange process:


48

1. Remote Sequence Number: The STATEL_INITIALIZE primitive returns the current Remote Sequence Number. If this hasn't changed since the last invocation of the primitive, the status of STATEL (VFS, messages) has not changed. This means that, as far as can be ascertained, data transmitted to the remote party in the previous session has not yet been used.

2. File attributes: The STATEL_STAT primitive can be used to obtain information on a specific VFS file. By checking the remote access sequence, the local application can determine whether the file has been accessed by the remote party.

3. File existence: The remote application can be designed to delete a file from the VFS after it has successfully processed the data contained in it. The local application can test for the presence of a file with a conventional name, and interpret its absence accordingly.

Keeping application files in the VFS It is possible to keep application related data permanently in the VFS, as a local copy of the VFS files is maintained; there is a small price to be paid for the STATEL calls, but this is really quite acceptable. If on-line help files, or parameter files, are kept in the VFS, they could be updated remotely by a server process.

Data security STATEL can attach a security code (key) to the files and directories in the VFS. In order to gain access to files protected with a key, it must be presented to STATEL_OPEN; this allows the application designer to lock the application directory and files.

CAPTURE - a data capture program In this section, we're describing an embryonic traffic ticket data capture program. We'll need a remote client program to be able to enter new information, and to consult and edit data previously entered. The host (the server) validates this data, and notifies the sending station of any errors or inconsistencies. When the server is happy with the data, it is consumed and can no longer be edited by the submitting station.

Please note that this example program uses the STATEL_EXEC() primitive, which (although quite useful) hasn't found favour with the developers using STATEL. Together with the perceived security risks associated with remote execution, this lack of popularity has ensured that the function has been declared obsolescent. Release 4.0 still offers this functionality, but it might be removed from future releases. It is important to stress that the STATEL_EXEC function does not present a security risk when properly configured, but that its use of external programs could indeed compromise the security of the system executing the programs.


49

Data Flow As you can see in Figure 11, the data capture process consists of two stages. First, we take the existing input data (here the traffic tickets), and use CAPTURE to enter it into the raw Traffic ticket data file. This file is processed by the remote server, and if any errors or exceptions are found, they are written to the Errors and exceptions file, and a message is sent to the data capture program that an edit operation is required. If so, the file is re-used by the local CAPTURE program to prepare the Edited Traffic ticket data file. This file is then integrated in the remote ticket database. The cycle of verification and edit is repeated as often as required.

The files used by CAPTURE are all located in the STATEL VFS, and processed in place. This allows us to take advantage of the efficient transmission of updates to existing STATEL files.

Figure 11: CAPTURE - Data Flow Diagram

The client program The most important aspect of this application is that it directly uses files in the VFS, rather than merely preparing the file in the local file system, and sending it to the remote host as a separate action. This is a key concept within STATEL, and should be understood and used to properly exploit the functionality of the


50

STATEL system. Doing this avoids redundancy both in local file space and in transmission cost and time.

Directories and files CAPTURE uses the STATEL subdirectory CAPTURE to store all its STATEL data files. The administrator (at the server side) has already created the VFS subdirectory by issuing the command

server[12] vfsmkdir -k biGbuCks police1+/CAPTURE

To stop other STATEL users from gaining access to the CAPTURE data (traffic tickets, and hence very sensitive data), he has used a key so that all subsequent access requests will need to know this key. He has also configured this key in the CAPTURE configuration files he sent to the data entry stations. The STATEL file name for the Traffic ticket data file is (not very originally) Traffic_ticket_data, and the Errors and exceptions file is called ERRORS.

Processing The CAPTURE data entry program uses the the presence of the ERRORS file in the VFS to decide to enter edit mode. The following code fragment can be used to verify the status of the files in the VFS:

int determineProcessingType(char *key) { struct stat errorStatus; struct stat dataStatus; int errorExists = STATEL_STAT("/CAPTURE/ERRORS", key, &errorStatus); int dataExists = STATEL_STAT("/CAPTURE/Traffic_ticket_data", key, &dataStatus); if (errorExists == 0) { if (dataExists == 0) { /* Files both present */ /* Is the error file a result of the current */ /* data file? */ if (errorStatus->remoteAccessSequence >= dataStatus->remoteAccessSequence) { /* Errors in previous data file */ return EDIT_DATA; } else { /* Error file is older - data file was processed OK */ return NO_DATA_FILE; } } else { /* Error file but no data file */ return INTERNAL_ERROR; } } else { /* No ERRORS file */ if (dataExists == 0) {


51

/* data file exists */ return NO_ERROR_FILE; } else { /* no data file yet */ return NO_DATA_FILE; } } }

After checking the status of the files, the program elects to open the data (and possible error) file:

switch (determineProcessingType(key)) { case INTERNAL_ERROR: fprintf(stderr, "Internal Error\n"); exit (1); case EDIT_FILE: /* data and error file - enter edit mode */ errorFile = STATEL_OPEN("/CAPTURE/ERRORS", READ, key); dataFile = STATEL_OPEN("/CAPTURE/Traffic_ticket_data", RDWR, key); EditTicketData(dataFile, errorFile); /* Start edit mode */ STATEL_CLOSE(dataFile); STATEL_CLOSE(errorFile); exit (0); case NO_DATA_FILE: /* no error - enter fresh catch */ dataFile = STATEL_OPEN("/CAPTURE/Traffic_ticket_data", WRITE, key); LoadTicketData(dataFile); /* Start data entry mode */ STATEL_CLOSE(dataFile); exit(0); case NO_ERROR_FILE: /* data exists - append new data */ dataFile = STATEL_OPEN("/CAPTURE/Traffic_ticket_data", APPEND, key); LoadTicketData(dataFile); /* Start data entry mode */ STATEL_CLOSE(dataFile); exit(0); }

Now that we have adequately opened the files, we can proceed with the interactive part of CAPTURE. The function EditTicketData() merits closer description, as it reads data from a VFS file (ERRORS), and uses it to determine which record in the data file to edit. This file is also stored in the VFS, which shows that an application (or part of it) need not use local files. In addition, the amount of data transferred between the two parties is minimised because only the edited data is sent to the remote party.

void EditTicketData(dataFile, errorFile) int dataFile; int errorFile; { ErrorRecordType errorRecord; DataRecordType dataRecord; long dataOffset; /* Read the records of the error file */ while (STATEL_READ(errorFile, &errorRecord, sizeof(ErrorRecordType)) > 0) { /* Locate and read the erroneous data record */


52

dataOffset = atol(errorRecord.recordNumber) * sizeof(dataRecord); STATEL_SEEK(dataFile, dataOffset, SEEK_SET); STATEL_READ(dataFile, &dataRecord, sizeof(DataRecordType)); /* Call the record edit routine (on screen) */ if (EditTicketRecord(&dataRecord) == CHANGED) { /* The record was changed - write it back to the VFS file */ STATEL_SEEK(dataFile, dataOffset, SEEK_SET); if (STATEL_WRITE(dataFile, ticketRecord, sizeof(ticketRecord)) != sizeof(ticketRecord)) { /* cannot write the record */ uError(0, "STATEL_WRITE", "Traffic_ticket_data"); /* DO NOT FORGET STATEL_TERMINATE() BEFORE exit()s */ STATEL_TERMINATE(); /* closes all open VFS files */ exit (1); } } } /* while */

Please note the use of atol() to determine the location of the data file record; you are strongly discouraged from using binary data in files which are used by both parties. A portable convention, such as 7-bit ASCII and zero-delimited strings, XDR, or ASN-1, should be used to ensure correct interpretation on both hosts.

The server program This program consists of two main services, i.e. the data validation and the database load services. The remote execution facility of STATEL is used to launch the program on the host machine, thereby eliminating the need to poll for arriving files.

/* *** Set parameters for remote execution */ remote_argv[0] = "/CAPTURE/Traffic_ticket_data"; remote_argv[1] = "NEW"; /* or EDITED when re-submitting */ remote_argv[2] = (char *)0; /* *** Redirect standard I/O files */ remote_stdio[0] = (char *)0; remote_stdio[1] = "/CAPTURE/stdout"; remote_stdio[2] = "/CAPTURE/stderr"; /* *** Execute the server */ if (STATEL_EXEC("ticket_server", remote_argv, remote_stdio) != 0) { /* Does not return remote status - see /CAPTURE/stderr!!! */ uError(0, ProgName, "on STATEL_EXEC"); STATEL_TERMINATE(); exit(3); }

The server program processes the STATEL file it obtains as its first argument, and treats it as a NEW or EDITED file based on the second argument. Note that the server does not need to bother with the identity of the client requesting the service, as all housekeeping is handled by STATEL. If it does need to know the identity of the remote party (for instance to send a reply later on, after this specific execution), it can read the contents of the STREMOTE environment variable. This variable contains the STATEL Nick Name of the remote party. In order to execute the remote execution request, it is sufficient to trigger the


53

execution of the incorporationAgent. It is of course possible to execute STATEL_INITIALIZE in a STATEL aware application; alternatively, one of the service programs can be launched. The server program needs only to keep track of the SNNs of the remote parties using the CAPTURE system, as the STATEL model only offers bi-directional, peer-to-peer services on a one-to-one basis. The overall structure of the server would be to call STATEL_INITIALIZE for each and every of the SNNs it handles.

S T A T E L P D U S P E C I F I C A T I O N S

55

STATEL PDU specifications The STATEL Protocol Data Unit (PDU) is used to exchange information between remote parties. It contains all the necessary instructions to bring the remote VFS into line with local changes, and to allow the remote party to receive messages. In addition, it contains information on the originator, recipient and status of the dialog.

Syntax The PDU is contains a number of components, which are generated by the use of the STATEL primitives. The basic format of a component is as follows:

<line-number>< ><category><.><type><(><parameter><,><parameter>...<)> Example: 1 ISC.ID(Eurostat,CBS-Heerlen,1,1,4.0.2,2)

The line number starts at 0 (protocol lines interpreted by the TM), and increments by one for each new line. The PDU components consist of 7-bit USASCII characters only; the exception to this rule are the file and message contents, which consist of 8-bit bytes (octets). When this type of data is present, it is always preceded by a count, so that they can be skipped. Table 11 gives an overview of the possible PDU components.

The latest release 4.5.0 supports PDUs expressed in XML where each component is formed as follows: <category.type cl=”line-number”> (parameter,parameter...)</category.type>

Example: <ISC.ID cl=”1”>(Eurostat,CBS-Heerlen,1,1,4.5.0,2)</ISC.ID>

In the rest of this document the first format will be used.

Chapter

5

P D U S P E C I F I C A T I O N S

56

Cat. Type Parameters Comments Protocol lines XFR IN Transfer Module information Communicates protocol related

information to the TM. These lines are ignored by the API.

Remote Logging Support LOG MS Message Transmits a status or error

message to the remote party. LOG RP Message Transmits (part of) a report to

the remote party. ADM RQ reportType Requests from a remote SNN

extended system reports. Allowed to be sent only from Eurostat

ADM RP gmtTime, targetFilename, totalLength, offset, Length, octet-1, octet-2, …

Transmits configuration and log files and saves them in the physical file system.

ADM LOG

XQ, XP XP

Placeholders for files extracted from the PDU before the normal incorporation process has been executed.

Internal STATEL Commands ISC ID localSNN,remoteSNN,localSeq,

remoteSeq,version, generation Uniquely identifies the PDU and implicitly acknowledges preceding PDUs.

ISC TR localSeq Terminates the PDU ISC MS type,length,octet-1octet-2... Message type and body ISC MP Sequence Informs the remote party that a

PDU cannot be retransmitted ISC RX start,end Requests the retransmission of a

PDU ISC RS preserveContents, Generation, DH_P,

publicKey Reset the sequence numbers and negotiate a new shared secret

ISC KE preserveContents, publicKey Acknowledge the negotiation and transmit the new public key

ISC EN nextComp, origSize, compEncSize, octet-1 octect-2

Encrypted PDU payload components

File Management Requests FMR CF VFS-path,type,key Creates a file/directory in the

VFS FMR DF VFS-path,type Deletes a file/directory from the

VFS FMR UF VFS-path,offset,length,octet-1octet-2... Enters data in a file FMR FS VFS-path,VFS-newname,

updateSeq,accessSeq Changes file attributes

FMR RL VFS-path Releases a file. Used to block partial files when PDUs are limited in size.

FMR UXCX DX RX

Placeholders for files extracted from the PDU before the normal incorporation process has been executed.


57

Remote Command Execution RCE LI command number,command name Prepares the remote execution RCE SR command number,standard input,

standard output,standard error Redirects files

RCE CP command number,parameter number, parameter

Defines a command parameter

RCE EX command number Starts execution Table 11: Overview of the STATEL PDU Components

One or more carriage-returns and/or line-feeds can be present after the closing parenthesis; they are simply ignored when the PDU is processed by the remote party.

Protocol lines These components contain protocol specific information, and are destined for the TM. The API module ignores them completely, and the line number is always zero (0). Because they are intended for the TM, they are obviously protocol specific. The contents of these lines are determined by the entries in the [Protocol lines] section of the tuple configuration file CF.

Internal STATEL Commands ISC.ID This component serves to identify the PDU. It should always have line number one (1), hence the string 1 ISC.ID( can be used as signature for STATEL PDUs. It contains the following entries:

• Source STATEL Nick Name

• Destination STATEL Nick Name

• Local Sequence Number, the unique sequence number for this PDU. If a duplicate sequence number is detected, the PDU can be safely ignored.

• Last Remote Sequence Number, the sequence number of the last remote PDU processed by the local party. All PDUs with this or earlier sequence numbers can be deleted from sendTransitDirectory.

• Protocol version number, to enable the processing installation to keep track of the versions installed at the remote site, and to take specific actions based on this version number.

• Generation of the PDUs. Each time STATEL_RESET() is executed, the generation number increases by one.

1 ISC.ID(SrcSNN,destSNN,LocSeq,RemSeq{{,version},generation}) Format


58

ISC.TR The ISC.TR component is always the last component in a PDU. The PDU's sequence number is repeated.

n ISC.TR(SourceSNN)

ISC.MS Messages (generated through STATEL_PUTMESSAGE) are transmitted with this component, which contains 8-bit data. To allow the use of 8-bit data, the length of the message body (in 7-bit USASCII) precedes the data block. Unlike the file data, the message data is not encrypted.

n ISC.MS(type,length,octet-1...octet-n)

ISC.RX This component is used to signal a missing PDU to the remote party. It can be followed by other components; if these would contain references to elements of the VFS which have been deleted by the other party, they will be ignored.

n ISC.RX(sequenceNumber,nextReceivedSequence)

The nextReceivedSequence is used to determine which PDUs need to be retransmitted by indicating to the remote party the next PDU which was received (and hence caused the sequence break to be diagnosed). Logically, all PDUs up to but not including this one, need to be retransmitted.

ISC.MP When a PDU cannot be retransmitted (because it is no longer present in the transmitTransitDirectory or has been damaged), this component is sent instead. Upon reception, the PDU with the corresponding sequence number is skipped. Any data contained in the PDU will not be present on the remote system.

n ISC.MP(sequenceNumber) ISC.RS To reset the sequence numbers and re-negotiate a shared secret using the Diffie-Hellman key agreement algorithm, the STATEL_RESET call constructs an ISC.RS component. The parameters are:

• The preserveContents flag, to indicate whether the contents of the VFS should be erased or maintained.

• The PDU Generation counter, which is incremented with each reset. PDUs that do not belong to the current generation are dropped.

• The new Diffie-Hellman ‘P’ parameter.

Format

Format

Format

Format


59

• The new Diffie-Hellman Public Key.

The PDU Generation of the emitting installation is kept to the current one until the ISC.KE component is received in return. Equally, the sequence numbers are maintained until the ISC.KE is received, guaranteeing that a retransmit request will not result in incorrect data. After the ISC.RS has been emitted, the installation is locked pending the receipt of the new public key.

n ISC.RS(0,3,0123456789ABCDEF0123…,0123456789ABCDEF0123…)

ISC.KE This component is returned to confirm the reset and to communicate the new Diffie-Helman public key to the installation that emitted the ISC.RS. It contains the preserveContents flag and the public key as a hexadecimal number. Using this remote public key, and the new local public and private keys, the shared secret is computed. Please note that the party that issued the ISC.KE has already computed the shared secret, as all the required information is contained in the ISC.RS.

n ISC.KE(0,0123456789ABCDEF1023…)

ISC.EN The ISC.EN component contains the compressed, encrypted representation of all the non-envelope (ISC, XFR) components of a PDU as a variable length stream of octects.

n ISC.EN(nextComp,origLength,compEncLen,contentOctets…)

The nextComp parameter enables the decoding routines to know the component number of the final envelope components without counting them in the decrypted and expanded ISC.EN data. The origLength and compEncLen are necessary to efficiently allocate the decompression and decryption buffers. compEncLen is also used to skip the data octets when scanning PDUs in the transit phases.

File Management Requests These components all serve to instruct the remote party to effect changes to his copy of the VFS. They are only written to the PDU after they have been validated through their application to the local copy of the VFS.

FMR.CF Create file in the VFS.

n FMR.CF(pathName,type,key)

type can be either F or W for a data file or D for a directory.

Format

Format

Format

Format


60

The component always contains the full, absolute path name. The concept of a working directory (cfr. STATEL_CHDIR) only exists inside the API; there are no components reflecting the setting or changing of this working directory inside the STATEL PDU.

FMR.DF This is the opposite of the previous component, and deletes a file or directory from the VFS. It is normally generated by the use of the STATEL_UNLINK primitive, but it can also result from calling STATEL_OPEN in WRITE mode on an existing file.

n FMR.DF(pathName,type)

If the file is a data file, the link count in the inode entry is decremented, and the entry in the directory is released (by changing the first character to a '\0'). If the link count becomes zero (0), the operating system file containing the data is removed. The pathName parameter is the absolute name of the VFS file.

FMR.UF This component is used to load data into a VFS file.

n FMR.UF(pathName,offset,length,octet-1...octet-n)

There is a one-to-one mapping of STATEL_WRITE calls issued by the application and the components in the PDU (so try to avoid writing single bytes to a VFS file if possible). The pathName parameter is again the absolute name of the VFS file; the execution of this component is not dependent upon a preceding open type component, and the interpreting process need not keep track of the status of the VFS files.

Please note that the data in the component is encrypted. It is worthwhile repeating that the data is encrypted upon entering the STATEL environment, and decrypted upon leaving STATEL. This ensures a modicum of security on unsafe networks and systems.

FMR.RL This component is used to signal that the remote application has finished processing a VFS file. Until this component is processed, the file is not accessible to the local applications. This is especially important when a file has been transmitted through several PDUs.

n FMR.RL(pathName)

Please note that there is no component indicating the execution of an STATEL_OPEN() call.

Format

Format

Format


61

FMR.FS When a file is accessed by the local party, this component is added to the PDU.

n FMR.FS(pathName,newName,modification,access)

where pathName is the absolute name of the VFS file, newName the (optional) new file name. modification and access indicate that the file has been changed or read during that sequence. They can be -1 when there has been no change. A component with both modification and access set to -1 is unusual but not to be considered an error (rather an expensive no-op).

FMR.CX, FMR.UX, FMR.DX and FMR.RX These components are generated through the use of the vfsextract service command. This command extracts files from PDUs that have not yet been incorporated (probably because they are stuck behind a hole in the PDU sequence). The incorporation programs skip these components, so that the file is not re-created when normal incorporation proceeds after the missing PDUs have been received.

Remote Command Execution This group of components supports remote command execution. Partial implementations need not support this functionality, and given its infrequent use, Remote Command Execution has been deemed obsolescent. Support for this feature might be removed from future releases.

RCE.LI Introduces the STATEL name of the command. There are tuple and system map files (CO) to convert this STATEL name to a name acceptable to the local system.

n RCE.LI(commandNumber,commandName)

RCE.SR This component indicates that the standard input, standard output and standard error files of the command should be redirected from/to VFS files. As in all PDU components, the path names are absolute.

n RCE.SR(commandNumber,input,output,error)

RCE.CP Every command parameter has its own component.

n RCE.CP(commandNumber,parameterNumber,parameterValue)

RCE.EX This command triggers the execution of the process.

Format

Format

Format

Format


62

n RCE.EX(commandNumber)

Limiting the size of a PDU Certain transmission services impose limits on the size of the items they transport. STATEL can adapt the size of the PDUs it generates to such a requirement through the use of the pduSizeLimit entry in the [Tuple Parameters] section of the tuple configuration file (CF). Whenever a PDU exceeds this limit, it is terminated and transmitted using the specified Transfer Module.

The PDUs can be transmitted when they have been completed, or at the end of the STATEL session. In the latter case, STATEL will restart the transmission process from the last successfully transmitted PDU onwards each time the STATEL_TERMINATE primitive is executed.

Interpreting the STATEL PDU The PDUs are interpreted in strict sequence number order, as actions in later PDUs can depend on the information contained in the earlier ones. If a gap in the sequence numbers (a sequence break) is found, an ISC.RX component is added to the next PDU sent to the remote party. Upon receiving an ISC.RX component, the local party retransmits all the PDUs between the sequenceNumber and nextReceivedSequence to the remote party. To avoid deadlock situations, the PDUs containing an ISC.RX request are detected by the receiving STATEL and the retransmissions are effected even when a sequence break was diagnosed. This avoids the deadlock situation where both parties have stopped processing incoming PDUs due to a sequence break and would never get to retransmitting the requested PDUs. A timer is used to limit the number of retransmission requests, and retransmissions. By default, a waiting period of eight (8) hours is respected before a retransmission is requested again. This counter is reset when (part of) the PDU gap is filled.

In order to enable STATEL to retransmit missing PDUs, these are only deleted from the sendTransitDirectory when they are acknowledged by the remote party through the ISC.ID component (the remoteSequence item).

The pduaccess API All the PDU access routines have been grouped into one library, affording a coherent approach to PDU management by the various STATEL subsystems (Hub, Gateway, Incorporation daemon, etc.).

The creation of PDUs isn’t covered by this library, but taken care of by the STATEL API module.

Format


63

There are four contexts available to the basic pduaccess routines. These contexts allow several individual PDUs to be opened concurrently, thus permitting the use of these routines in, for example, the PDU creation routine in the STATEL API (remember that the PDU creation routine needs to access existing PDUs that have not yet been transmitted). Context 0 is used for routine PDU processing by the incorporation subsystems. Context 1 is used by the PDU creation routine. Contexts 2 and 3 can be freely used.

PDU enumeration The pduTally() routine is used to scan a directory containing PDUs, and make a list of all the PDUs destined for a particular SNN. It always uses context 0 for its low-level access to PDUs.

int pduTally(char *SNN, int type);

where type can be either ST_PDU_SEND or ST_PDU_RECEIVE. The function returns the number of PDUs it was able to discover. Once a number of PDUs have been discovered (tallied), they can be opened in sequence with the pduOpen() routine.

This routine returns a sorted array of longs containing the PDU sequence numbers collected by pduTally(). The information is obtained from the same list used by pduOpen().

long *pduSequence();

This is an easy way to discover whether there are gaps in the sequence, without having to open each PDU in a loop. The call is overhead-free because the list is kept from pduTally (where it was required for the sort operation).

Opening and closing PDU files Setting the open modes PDUs can be opened in read-only mode when the only purpose is to inspect them. By default, the pduaccess routines use read-only mode; if read-write access is desired (for example when the shared secret is available and the PDU should be decrypted, service routines should be called to set the open mode to read-write, and to supply the shared secret.

This service routine sets the mode used to open the PDUs. It applies to both pduOpenByName and pduOpen (remember that pduOpen uses the services of pduOpenByName).

char *pduSetOpenMode(int context, char *mode);

where mode is one of the valid fopen() modes. These are available through STATEL as FO_(R,W,A){P}_(BINARY,TEXT), but the more usual strings such as “a+” can also be used.

Contexts

pduTally

Format

pduSequence

Format

pduSetOpenMode

Format


64

This routine should be used if the PDU has to be decrypted (and uncompressed) by the pduaccess library, to provide the decryption key. Please refer to the pduComponent() routine for details on the exact moment and method of decryption.

char *pduSetDecryptionKey(int context, char *secret);

Typically, the secret will be obtained from the tuple’s configuration file. If the key has not been loaded, encrypted PDUs will be left encrypted, and accessing the components will involve the non-encryptable components such as ISC.EN.

The pduOpen() call can only be used after a successful call to pduTally(), as it uses the list of PDUs established by this routine to allow in-sequence access to the PDUs available for an SNN. The pduOpen() routine needs to be given a context. Context 0 (used by pduTally()) can be safely reused.

long pduOpen(int context, int flag);

where flag can be either ST_PDU_NEXT, ST_PDU_PREVIOUS, ST_PDU_FIRST, or ST_PDU_LAST to access the obvious PDU in the list. Please note that the first use of ST_PDU_NEXT after a successful call to pduTally() is equivalent to using ST_PDU_FIRST.

This routine is used to open a specific PDU through its file name. It’s used by pduOpen(), to ensure a consistent approach to PDU access.

int pduOpenByName(int context, char* pduName);

The routine does not verify if the PDU is formally correct; subsequent calls to the component access routines might yield unpredictable results if a non-PDU is opened with this routine.

Once a PDU has been opened, it should be closed with this routine. Under no circumstances should the file itself be closed directly. To support the sequential processing of PDUs through the pduOpen() routine, the pduClose() routine can be used to delete the PDU (meaning that the routine never needs to obtain or manipulate the underlying file name).

void pduClose(int context, int flag);

where flag can be either ST_PDU_KEEP or ST_PDU_REMOVE. PDUs opened in read-only mode can be removed, so the user should make sure the correct close mode is provided.

Accessing PDU components The objective of opening a PDU is to read and process its components. The pduComponent routine steps through the PDU components, decrypting the ISC.EN component as it is encountered. The approach used to decryption and uncompression is the following:

pduSetDecryptionKey

Format

pduOpen

Format

pduOpenByName

Format

pduClose

Format


65

• When the ISC.EN component is encountered, the components following it are read and stored in a buffer.

• The decryption key is used to decrypt the ISC.EN payload.

• The original size provided in the ISC.EN component is used to uncompress the (now decrypted) payload.

• The decrypted and uncompressed payload (consisting of a number of PDU components such as FMR.UF) is written back to the PDU file

• The stored components are adapted (line numbers) and written back to the PDU file in their appropriate location

• The routine returns to the first non-read component, and presents it to the user.

The effect of this approach is that once the ISC.EN component has been read with a decryption key present, it ceases to exists even if the PDU is not processed and removed. The resulting decrypted and uncompressed PDU will in all likelihood be larger than the original version.

This function returns a pointer to a PduComponentBuf structure.

typedef struct PduComponentBuf_TAG { long seqNum; char category[4]; char entry[3]; int items; char **item; long offset; } PduComponentBuf;

Note that the interpretation of the items needs to be provided by the user - each item is stored in a NULL-terminated buffer, but is not guaranteed to be a NULL-terminated string.

You can't use this function to step backwards through the PDU components.

PduComponentBuf *pduComponent(int index, int flag)

Flag can be:

• ST_COMP_FIRST

• ST_COMP_NEXT

pduComponent

Format


66

Service functions A number of functions are available to set or access information on the open PDU. They are:

Name Description

FILE* pduFileHandle(int context) Returns the File handle of the currently open PDU

long pduPos(int index) Returns the current offset in the PDU file. Typically, this will reflect the offset of the next component.

long pduSize(int index) Returns the file size of the currently open PDU file.

char* pduFileName(int index) Returns the name of the currently open PDU file.

L O G G I N G F E A T U R E S

Chapter

6 Logging Features This chapter describes the logging features that are available in STATEL version 3.3 and upwards.

Logging of urgent events Introduction A number of events have been identified and are logged to a file for analysis. These events include:

• Transfer module errors

• Incorporation errors

• Retransmission requests sent to a remote party

• Retransmissions to a remote party

• Reception of duplicate PDUs

• Reception of damaged PDUs

• Configuration problems

• Skipping of damaged PDUs Please note that this is a purely local log function, and no data is transmitted to the remote party.

Activating the logging feature The logging is activated by specifying the name of the file in the SNN.ISF file, in the section [System Parameters]. The keyword is urgent Log File (see figure 31). It can be overridden for individual SNNs (in the [Tuple Parameters] section of the tuple's CF file).

# Generated by addtuple.c 3.9 on Tue Dec 6 21:43:12 1994 [Statel Nick Names] localSNN=install


68

remoteSNN=loop,loop remoteSNN=lu-ecc-1,lu-ecc-1 [System Parameters] badPDUDirectory=~/bad sendTransitDirectory=~/send receiveTransitDirectory=~/receive minimumDiskSpace=5000000 remote Report File=remote_report_file remote Diagnostics File=/tmp/remote_log_file urgent Log File = $STATELBASEDIR/urgent_log_file [Registered Users] User=sae

The file definition follows the usual STATEL conventions:

Specification Expansion first pass $VAR/name VAR is interpreted as an environment variable. @PAR@/name PAR is a parameter defined earlier (CF only) second pass ~/name name in the tuple's directory name name in the STATEL Base directory /name name without change Table 12: STATEL File name expansion rules

The definition of the remote Report File and the urgent Log File are equivalent, as the first one is a relative name (and hence placed in the STATEL Base directory), and the second one explicitly references it.

Size monitoring and backups The urgent log file is not limited in size, nor does it have a backup feature as do all the Transfer Module logs. The rationale here is that this log will be processed and maintained by external programs (or scripts), and in any case contains information that warrants prompt action by the STATEL administrator.

Format of the log entries The general layout of the entries in the urgent log is shown below:

Remote SNN:time:UR:local sequence:remote sequence:code:description

The time field is in seconds since the epoch, and can be converted with the STATEL utility spr. The UR field is present to allow the three logs to be written to the same file.

lu-ecc-1:836220314:UR:420:620:14:iscid:retransmission requested from 621 to 622 lu-ecc-1:836228429:UR:424:630:14:iscid:retransmission requested from 631 to 635 lu-ecc-1:836229055:UR:426:637:14:iscid:retransmission requested from 638 to 639 lu-ecc-1:836229111:UR:427:641:14:iscid:retransmission requested from 642 to 644 lu-ecc-1:836243741:UR:429:646:9:incorp:unlinked duplicated PDU 344 (U0000344) lu-ecc-1:836243741:UR:429:646:9:incorp:unlinked duplicated PDU 351 (U0000351) lu-ecc-1:836243741:UR:429:646:9:incorp:unlinked duplicated PDU 515 (U0000515) lu-ecc-1:836244865:UR:429:654:14:iscid:retransmission requested from 655 to 656

Description of the log entries This section describes the various urgent log entries and under which circumstances they are generated.


69

No Description Origin Cause 1 Updating deleted file

file incorp(FMR.UF) The file does not exist on the local system

but the remote tries to update it. 2 Releasing non-existing

file file incorp(FMR.RL) Could not find file to release.

3 Cannot find file incorp(FMR.FS) Could not find VFS file for status change. 4 Cannot open dir for

file incorp(FMR.FS) Could not open directory dir to rename

file. 5 Deleting non-existing

file file incorp(FMR.DF) The VFS file could not be deleted as it was

removed by the local party. 6 Error nnn creating F

file incorp(FMR.CF) An error occurred trying to create VFS file

(F) or directory (D) file. 7 cannot chdir to dir

errno x incorp() Configuration error: the receive transit

directory cannot be found. 8 cannot open directory

dir errno x incorp() Configuration error: the receive transit

directory cannot be accessed. 9 unlinked duplicated

PDU x (name) incorp() A duplicated PDU was received and was

deleted from the directory. 10 unlinked bad PDU x incorp() PDU x was incorrect (format) and was

removed from the directory. 11 copied & unlinked

bad PDU x incorp() PDU x was incorrect (format) and was

removed from the directory after being copied to the bad PDU directory.

12 cannot copy bad PDU x

incorp() PDU x was incorrect (format) and was not removed from the directory because it could not be copied to the bad PDU directory.

13 Cannot update timer entry in CF (error %ld)

incorp(ISC.ID) The CF file could not be updated with the new time-stamp.

14 retransmission requested from x to y

incorp() PDUs from x (inclusive) to y (non-inclusive) were re-requested.

15 retransmit x to y incorp() PDUs x (inclusive) to y (non-inclusive) have been retransmitted to the remote party.

16 *ERROR* TM(par1,par2) returned base+error

uAgent The TM could not effect the transfer and returned error code error.

Table 13: Overview of the urgent log entries

Support for logging of remote errors Introduction A number of errors occurring at a remote installation can be sent to the local STATEL installation.

Activating the feature On the remote system The facility is activated by specifying logToRemote=YES in the [Tuple Parameters] section of the CF file (it can be entered as a system default in the [System Parameters] section of the SNN.ISF file).


70

On the local system The entry remote Diagnostics File should be set to indicate where the log file should be stored. Please note that this entry can be specified in the [System Parameters] of SNN.ISF with an override in the [Tuple Parameters] section of the CF files.

Size monitoring and backups The remote diagnostics file is not limited in size, nor does it have a backup feature as do all the Transfer Module logs. The rationale here is that this log will be processed and maintained by external programs (or scripts), and in any case contains information that warrants some action by the STATEL administrator.

Format of the log entries The general layout of the entries in the remote diagnostic log file is below:

Remote SNN:time:MS:local sequence:remote sequence:code:description

The local sequence is relative to the logging SNN, not to the remote party producing the diagnostic entry.

Description of the log entries This section describes the various urgent log entries and under which circumstances they are generated.

No Description Origin Cause 1 Updating deleted file

file incorp(FMR.UF) The file does not exist on the local system

but the remote tries to update it. 2 Releasing non-

existing file file incorp(FMR.RL) Could not find file to release.

3 Cannot find file incorp(FMR.FS) Could not find VFS file file for status change.

4 Cannot open dir for file

incorp(FMR.FS) Could not open directory dir to rename file file.

5 Deleting non-existing file file

incorp(FMR.DF) The VFS file file could not be deleted as it was removed by the local party.

6 Error nnn creating F file

incorp(FMR.CF) An error occured trying to create VFS file (F) or directory (D) file.

Table 14: Overview of the remote diagnostics entries

Support for the transmission of reports The LOG.RP protocol component has been added to support the transmission of information from a remote party. These reports can be produced as separate executables, allowing various problems to be diagnosed. The latest version supports the automatic transmission of “one line” reports from remote SNNs to


71

Eurostat with details about the STATEL version they use, the operating system and the basic hardware configuration of the hosting machine.

Activating the feature The remote Report File keyword in the SNN.ISF or CF files can be used to define the location of the report file on a system-wide or per tuple basis.

Size monitoring and backups The remote report file is not limited in size, nor does it have a backup feature as do all the Transfer Module logs. The rationale here is that this log will be processed and maintained by external programs (or scripts), and in any case contains information that warrants some action by the STATEL administrator.

Format of the log entries The general layout of the entries in the remote report log file is as shown below:

remote-SNN:time:RP:report line

The format of the report line is wholly dependent upon the program used to produce the remote report. It is suggested that each tuple be assigned its own report file, as the output of reports could be rather voluminous.

Transmission of extended system reports The latest version supports the automatic transmission of extended system reports from remote SNNs to Eurostat, which includes the STATEL configuration files, the available log files of the Transfer Modules, list of the pending PDUs and information about the local software and hardware configuration.

C O M P I L I N G S T A T E L

Chapter

7 Building STATEL This chapter discusses the organisation of the STATEL code and how to compile it for the various supported platforms

Introduction Compiling STATEL is not difficult, but requires reasonable knowledge of the many components involved. Even though the STATEL Core is written in pure ‘C’, and didn’t require external features or libraries until the incorporation of OpenSSL for the encryption and zlib for the compression, a complete STATEL installation contains, in addition to the Core,

• Perl, Java and Tcl/Tk language bindings

• The STATEL Gateway (using ProFTPd and isect)

• Unix Command Line Utilities (requiring dllist)

• The Unix incorporation daemon (requiring dllist)

• The Windows STATEL Hub, requiring

• Windows Utilities such as the File Manager, the STATEL Robot Service, written with Borland C++ Builder.

The code is stored in a CVS repository.

Please note that unlike many packages distributed for Free Operating Systems such as Linux, STATEL is meant to be built and installed by a skilled computer professional.

The STATEL Core The STATEL Core is written in standard ‘C’, and because of its origins, it contains macros to ensure it compiles correctly on old, Kernighan & Ritchie type compilers as well as the current ANSI ‘C’ compilers. Some of the more recent additions have abandoned this practice, so please ensure a recent ‘C’ compiler is available. Since version 4.0, the STATEL Core requires the

Warning:


73

OpenSSL and zlib libraries to compile. It links statically to these libraries to ensure that neither OpenSSL nor zlib need to be installed on the production machine.

STATEL for Unix The Core has been compiled on the following platforms:

• Solaris 2.6, 7 8 and 9 with the GNU Compiler Collection (gcc) and Sun Forte 6.2

• IBM AIX 4.3.3 with the IBM ‘C’ compiler and gcc

• Hewlett-Packard HP-UX 11.0 with gcc

• Hewlett-Packard Tru64 5.1a with the Tru64 ‘C’ compiler

• Linux with gcc

Versions of gcc from 2.95 upwards have been used to compile version 4 of the Core.

The compilation process compiles not only the core, but the all the STATEL components for Windows, and the required supporting libraries.

Providing OpenSSL The STATEL build process doesn’t provide OpenSSL, as this is usually a system-wide facility with specific installation requirements. Even though STATEL only uses the cryptographic libraries, OpenSSL should be installed and configured before building STATEL.

Building process overview The STATEL CD contains a “Statel4” subdirectory with the latest version of the software. To build it, issue the ./configure command and answer the questions:

ultra~/Development/STATEL/Statel4[109] ./configure Please enter the base directory for the STATEL installation: /home/sae/statel Please enter the STATEL user [sae]: Please enter the STATEL group [sae is member of <staff sysadmin statel>]: statel Do you want debugging made available [Y]: Do you want STATEL Logging switched on [Y]: Where is the OpenSSL crypto library located [/usr/local/ssl/lib/libcrypto.a]? Where is the OpenSSL include directory located [/usr/local/ssl/include]? Do you want to use gcc or Sun's cc compiler? cc Please review the following configuration: =========================================== # Created on Thu Jan 8 09:17:27 MET 2004


74

VER=4.1.2 PREV = 4.0.0 PREFIX = /home/sae/statel LIBINSTALLDIR = "$(STATELLIB)" BININSTALLDIR = "$(STATELBIN)" STATELUSER = sae STATELGROUP = statel STATELINCLUDE = ../Include FEATURES = -DDEBUG -DST_STATISTICS -D__STATEL_TM__ CRYPTOLIB = /usr/local/ssl/lib/libcrypto.a COMPRESSLIB = ../zlib/libz.a EXTRALIBS = -L ../dllist CRYPTOINCLUDE = -I /usr/local/ssl/include DLLISTINCLUDE = -I ../dllist COMPRESSINCLUDE = -I ../zlib ISECTINCLUDE = -I ../isect EXTRAINCLUDES = $(DLLISTINCLUDE) $(CRYPTOINCLUDE) (COMPRESSINCLUDE) $(ISECTINCLUDE) MACHINE = solaris CC = cc BASECFLAGS = -g -Dsolaris -DHAS_FTRUNCATE SHLIBFLAGS = -Kpic SHLIBLINK = -G SHAREDREF = -R$(LIBINSTALLDIR) SOCKETLIBS = -lsocket –lnsl DEPENDFLAG = -xM1 SYMLINKTEST = -h DEFS = -DHAS_UNISTD_H -DSTDC_HEADERS -DHAS_SGTTY_H HFLAGS = -I $(STATELINCLUDE) LFLAGS = $(SHAREDREF) -L $(STATELLIB) $(EXTRALIBS) CFLAGS = $(BASECFLAGS) $(HFLAGS) $(FEATURES) -D$(MACHINE) $(DEFS) $(EXTRAINCLUDES) # RULES .o.a: @$(AR) $(ARFLAGS) $@ $*.o ===========================================

The command recognises the following environment variables, and prompts for them if they are not provided:

Variable Description

STATELBIN The location of the STATEL binaries

STATELLIB The location of the STATEL libraries

PREFIX The overall location of the STATEL directory hierarchy

STATELUSER The user that will own the STATEL files

STATELGROUP The group to which the STATEL files will belong; also the group that will be used to communicate between the user environment and the STATEL environment

When the generated configuration file is deemed acceptable, STATEL can be built by issuing the command “make”. The supplied Makefile also installs the software in the directory pointed to by the environment variable PREFIX (cfr. the prompt above).


75

Compiling zlib Please review the Makefile for zlib, as the compiler flags can influence the speed of the compression and decompression quite significantly. Make sure the code is sufficiently optimised. The configure script should also be adopted if an already installed version of zlib should be used.

STATEL Core for Windows The Core is compiled with Microsoft’s Visual C++, but it can also be compiled with Borland’s (free) C++ compiler. A dump of the current build directory is provided on the CD. Given the limitations of the Visual C++ IDE, new projects might have to be created, or the existing directory names would have to be changed.

Perl language bindings The Perl language bindings are created using the same compilers as the STATEL Core, and the desired version of Perl. They are implemented as a standard Perl XS module, and should be installed in the appropriate location in the Perl module repository.

Java language bindings The interface between Java and STATEL is realised as two classes, that mimic the Java “RandomAccessFile” class. The link between the STATEL shared libraries (.so or .DLL) is handled by a special library, written in ‘C’ and compiled with the same compilers as the Core.

Tcl/Tk language bindings These are implemented as a ‘C’ library loaded into the Tcl/Tk interpreter. It is the man they installation of Tcl/Tk.

STATEL Command Line Utilities The STATEL Command Line Utilities require dllist, a public domain double linked list library, which is also used by the stincdplus incorporation daemon. They can be compiled for Unix and Windows, but until now there has been no demand for the Windows version. The dllist library hasn’t been ported to Windows yet.

technical manual - europa · technical manual . application-to-application data exchange toolkit...

Documents