commerce engine adapter developer's guide

Commerce Engine Adapter Developer's Guide

for

Singl.eView Convergent Billing v6.00

Publication Date November 2004

Document Number SVDOC-55-1068-BR3, v2.0

Disclaimer Although every effort has been made to ensure the accuracy of the information contained in this document, Intec Billing Ireland ('Intec') makes no warranty, expressed or implied, with respect to the quality, correctness, reliability, currency, accuracy, or freedom from error of this document or the products it describes. Intec may make improvements and/or changes to the products and services described in this document at any time. Unless otherwise explicitly agreed in the licence agreement governing use of this documentation, Intec disclaims all liability for any direct, indirect, incidental, consequential, special, or exemplary damages resulting from the use of the information in this document or from the use of any products described in this document. Mention of any other product not owned by Intec does not constitute an endorsement of that product by Intec, and if such products are provided by Intec, they are provided on an 'as-is' basis without warranty of any kind from Intec. Data used in examples and sample data files are intended to be fictional and any resemblance to real persons or companies is entirely coincidental.

Persons reading this document should rely on their own enquiries in making any decisions touching on their own interests.

Licence Agreement The software described in this guide is supplied under a licence agreement and may only be used in accordance with the terms of that agreement.

This documentation and products described in this document are provided under licence from Intec. It is possible that the applicable licence does not cover all the elements of functionality described in this document. If you have any questions regarding the functionality for which your site is licensed, contact your contract administrator. Confidentiality obligations set forth in the licence apply to all documentation that is marked as Intec Confidential Information.

Copyright Information © 2004 Intec Billing Ireland (An Intec Telecom Systems PLC Group Company). All rights reserved. Apart from any use permitted under the Copyright Act, no part may be reproduced by any process without the written permission of Intec Billing Ireland, IDA Business Park, Dangan, Galway, Ireland.

The following is a trademark of Intec Billing Ireland:

The following are third party trademarks:

Microsoft is a registered trademark and Windows is a trademark of Microsoft Corporation in the USA and other countries. FrontPage is a trademark of Microsoft. TEX is a trademark of the American Mathematical Society. PostScript and Acrobat Reader are trademarks of Adobe Systems Incorporated. UNIX is a registered trademark of The Open Group. Tuxedo is a registered trademark of BEA Systems, Inc. Oracle is a registered trademark of Oracle Corporation. AIX is a registered trademark of International Business Machines Corporation. InstallShield is a registered trademark of InstallShield Software Corporation. Java is a trademark of Sun Microsystems, Inc. Hyperion SQR and Hyperion Intelligence are trademarks of Hyperion Solutions Corporation.

Table of Contents

Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00 i

Preface......................................................................................................................................iii Welcome to Singl.eView Convergent Billing ...........................................................................iii Who Should Read this Guide .................................................................................................... iv Before Beginning....................................................................................................................... iv How this Guide is Structured...................................................................................................... v Other Singl.eView Convergent Billing Documentation ............................................................. v Conventions Used in this Guide ................................................................................................ vi

Chapter 1 � Overview...........................................................................................................1-1

Chapter 2 � Protocol.............................................................................................................2-1 External Issues .........................................................................................................................2-1 Design Issues ...........................................................................................................................2-2 Failure Handling Mechanisms .................................................................................................2-4 Transport Issues .......................................................................................................................2-5

Latency........................................................................................................................2-5 Delivery Guarantees....................................................................................................2-6 Addressing ..................................................................................................................2-6 Load Balancing ...........................................................................................................2-7

Example Protocol Design for Service Authorisations .............................................................2-8 Service Authorisation Request Lost............................................................................2-9 Service Authorisation Response Lost .......................................................................2-10 Request Processing Exceeds Timeout.......................................................................2-10 Service Controller Failure .........................................................................................2-10 Failover of Commerce Engine ..................................................................................2-11

Example Protocol Design for Usage Debits ..........................................................................2-12 Debit Usage Request Lost .........................................................................................2-12 Debit Usage Confirmation Lost ................................................................................2-13 Debit Usage Confirmation Timed Out ......................................................................2-14 Service Controller Failure .........................................................................................2-14 Failover of Commerce Engine ..................................................................................2-15

Chapter 3 � Adapter Design.................................................................................................3-1 Commerce Engine Interface Functions....................................................................................3-1 Interaction Models ...................................................................................................................3-2 Asynchronous Design..............................................................................................................3-3

Advantages of Asynchronous Design .........................................................................3-4 Disadvantages of Asynchronous Design.....................................................................3-5

Synchronous Design ................................................................................................................3-6 Advantages of Synchronous Design ...........................................................................3-8 Disadvantages of Synchronous Design .......................................................................3-8

Chapter 4 � Failover .............................................................................................................4-1 IP Address Failover .................................................................................................................4-1 Multicast ..................................................................................................................................4-3 Controller Managed.................................................................................................................4-4

Table of Contents

ii Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00

Chapter 5 � Management Interface ....................................................................................5-1

Index

Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00 iii

Preface

Welcome to Singl.eView Convergent Billing

Singl.eView Convergent Billing has been designed specifically for new generation customer care and billing environments, to increase competitive advantage by making it easier to design, deliver, and bill the products and services customers want. Singl.eView Convergent Billing provides modules for rating, discounting, and bill production.

Singl.eView Convergent Billing is based on industry standard architecture, relational database technology and interfaces, to ensure portability across operating platforms.

Singl.eView Convergent Billing offers service providers:

• Increase in competitive advantage by making it easier to design, deliver, and bill products and services.

• Hardware-efficient, real-time architecture that can be scaled as they grow through database distribution.

• Expression-driven design enabling solutions that can grow with changing business models (due to either company growth or technological advances).

• Comprehensive security features that provide control of access to customer information.

• Elimination of any single point of failure of the rating engine, and transaction services when installed with the high availability configuration (SV HA Config.).

• The ability to distribute the application for scalability of rating, billing transaction services, reports, and batch interfaces.

• Support for international implementations including Unicode.

This guide details the high-level concepts for developing Commerce Engine adapters for Singl.eView Convergent Billing.

Preface

iv Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00

Who Should Read this Guide

This guide is specifically aimed at system architects and developers when deploying the Commerce Engine.

Before Beginning

This guide assumes that the reader has some experience with using:

• Singl.eView Convergent Billing

• The Commerce Engine concepts and configuration

• TRE server concepts and configuration

• Tuxedo concepts and configuration

• Networking protocols

The reader should also be familiar with the concepts described in the High Availability and Scalability Guide for Singl.eView Convergent Billing.

Preface

Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00 v

How this Guide is Structured

This guide consists of the following five chapters:

• Chapter 1, Overview, describes the high level concepts of developing Commerce Engine adapters for Singl.eView Convergent Billing.

• Chapter 2, Protocol, provides guidelines and suggestions for ensuring that the protocol used or designed for use with the Commerce Engine meets the design criteria.

• Chapter 3, Adapter Design, describes the high-level design choices for the adapter and their performance and useability characteristics.

• Chapter 4, Failover, describes the alternate mechanisms that can be used to allow the service controller to continue operating when a failover occurs.

• Chapter 5, Management Interface, describes the management interface requirements for the Commerce Engine adapter.

Other Singl.eView Convergent Billing Documentation

This guide is part of a suite of Singl.eView Convergent Billing documentation, covering every aspect of the system. For detailed information about the documents available, refer to the Overview for Singl.eView Convergent Billing.

In particular, the following manuals can be of assistance in developing Commerce Engine adapters:

• Tuxedo Guide for Singl.eView Convergent Billing

• TIBCO Adapter Guide for Singl.eView Convergent Billing

• System Administration Guide for Singl.eView Convergent Billing.

• High Availability and Scalability Guide for Singl.eView Convergent Billing

Preface

vi Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00

Conventions Used in this Guide

The following conventions are used in this guide for product names:

Full Name Abbreviation Singl.eView All versions of the Singl.eView

product

Singl.eView Convergent Billing Convergent Billing

SV HA Config. SV HA Config.

The following conventions are used in this guide for text presentation:

Style Description

File ➩➩➩➩ Exit Indicates that the File menu should be opened and the Exit option selected.

Save button Button on the toolbar or on a form.

Customer Detail form Name of a specific form.

Account Status field Name of a field on a form.

Unix output In interactive and code examples, this typeface is used to indicate system output.

Unix input In interactive examples, this bold typeface is used to show user input.

Ctrl+F4 In procedures, a key sequence indicates that the Ctrl key must be held down while another key is pressed. In this example, the Ctrl key must be held down while the function key F4 is pressed.

<variable> Indicates that the value of the parenthesis is a variable.

(cont) Indicates the UNIX output, UNIX input, or code fragment continues on a single line, but is broken to fit on the printed page.

Use of a mouse in Singl.eView GUI applications is identical to that in other Microsoft Windows applications.

Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00 1-1

Chapter 1 Overview

This document describes guidelines for the development of adapters that provide an interface to the Singl.eView Commerce Engine ('Commerce Engine') service authorisation functions in a highly available environment.

The Commerce Engine is intended for deployment in real-time networks to support prepaid billing. In this context, the Commerce Engine services must be highly available and have low latency to ensure uninterrupted service and avoid service establishment delays. The overall system must also ensure that both the consumer and the service provider are protected from financial risk when failures do occur.

SV HA Config. provides a framework for ensuring that the Commerce Engine services are highly available and have adequate performance. These services are provided as Transaction Engine (TRE) interfaces, and usually require an interface adapter compatible with the network or other equipment accessing the Commerce Engine services. This adapter must complement the SV HA Config., providing appropriate interfaces for management, supporting the high availability approach associated with the configuration, and adding minimal latency to Commerce Engine transactions.

Prebuilt adapters exist for interaction with some network elements and transport protocols (for example, Alcatel SCP over TIBCO Rendezvous and Cisco SSG using the Radius UDP protocol with extensions), but there are many platforms for which adapters will need to be built. When building these adapters, the following three key design criteria must be taken into account:

• Protocol

The protocol used to communicate between the Commerce Engine and a network element is crucial to achieving high availability and acceptable performance. In particular, the protocol must clearly define roles and responsibilities involved in network and component failures.

• Adapter design

A number of mechanisms can be used to access TRE functionality and the choice of these mechanisms can have a significant effect on performance and flexibility.

Overview

1-2 Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00

• Management interface

To work effectively with SV HA Config., an adapter must provide a management interface allowing it to participate in failover and monitoring mechanisms.

Subsequent chapters of this document describe each of these criteria in more detail, and provide a technical basis on which to design and build a Commerce Engine adapter.


Chapter 2 Protocol

The protocol used to communicate between a network element and the Commerce Engine has a significant impact on the availability and performance of the service being provided to consumers. The protocol must be efficient, capable of supporting large volumes of concurrent transactions, and in particular, must clearly define roles and responsibilities involved in network and component failures. This chapter provides guidelines and suggestions for ensuring that the protocol used or designed for use with the Commerce Engine meets the design criteria.

External Issues

The protocol design needs to be undertaken in conjunction with the vendor of the network element or service component responsible for service establishment and control (for example, an intelligent network service control point or a filtering router such as the Cisco SSG). These network components sometimes have predefined or base protocols that constrain the implementation. These components are referred to as service controllers in this guide.

In many implementations, it might be necessary to support interaction with multiple vendor network elements; therefore, either a shared protocol or multiple adapters must be built. For network elements or service components with lesser latency and availability requirements, it might be sufficient to use one of the Singl.eView EAI adapters or the Java interface. This choice should not be made without careful analysis.

The service provider might also have specific requirements or standards compliance directives that influence the protocol design. In some cases these requirements can reduce the availability and increase the financial risk to consumers or the service provider. This must be made clear to the service provider.

Protocol


Design Issues

The Commerce Engine service has the following two basic interactions that must be supported by the protocol:

• Authorise a service request

This ensures that the customer has sufficient account balance and access to the service, then reserves a balance for projected usage charges.

• Debit an account for service usage

This ensures that reservations related to service authorisation and usage are removed or decremented, and replaced with actual usage charges.

Figure 2�1 depicts a typical service session, with the service controller obtaining an initial authorisation of units (for example, minutes), followed by an additional authorisation when the initial units are used, and then sending a final debit request when the service session completes.

Service request (eg. dial Authorise

N units

Service ready (eg. call

Service usage complete (eg. hang

Debit for

Usage

Service user Servicecontroller Adapter

Authorise

N units

Debit for

Usage

CommerceEngine

Authorise service (N units

(N+M) units

Authorise service (N units

(N+M) units

N unit timer

M unit timer

Figure 2�1 Typical Service Session

Protocol


Both the authorisation and debit actions require a request from the service controller and a reply to the service controller. The semantics of these actions within Commerce Engine are determined by configuration. Regardless of that configuration, a number of modes of failure must be accounted for in the protocol or configuration:

• A request can be lost in transit.

• The active Commerce Engine instance might fail during processing a request without committing changes.

• The active Commerce Engine instance might fail during processing a request after committing changes.

• The active Commerce Engine might not complete processing of a request within a predefined maximum time period.

• A response can be lost in transit.

• The service controller might fail during request processing.

It is important to remember that all of these failures are possible and must be handled by the protocol and endpoints involved. Hardware, network, and protocol redundancy can make such failures highly unlikely and it might be acceptable to ignore the failures, but the protocol must still recognise those failures. Assuming that such failures will not occur is dangerous and could result in unexpected failures and significant financial risk.

It is also important to recognise that the service controller cannot reliably distinguish between the first five failures above; the service controller sends a request and waits for a response. Transport-level mechanisms can be used to detect communication failures, but these mechanisms have overheads that increase latency and are not generally guaranteed: the better the guarantee, the higher the overhead. In all cases, the service controller must rely on a timeout to detect failures not caught by other mechanisms.

Each of these modes of failure need to be addressed separately for service authorisations and usage debits, since they have different latency and availability requirements, and associated financial risks. Information on how these can be addressed by the protocol and endpoints for an example configuration is contained in Example Protocol Design for Service Authorisations and Example Protocol Design for Usage Debits later in this chapter.

Protocol


In addition to the authorisation and debit interactions introduced at the beginning of this section, some implementations require additional interactions (for example, advice of charge or account balance queries). The same modes of failure apply to these interactions and should have a similar analysis applied. It is expected that these additional services would be read only, so the likely implications of such failures are less critical.

Failure Handling Mechanisms

There are a number of generic failure handling approaches with distinct characteristics. All interactions are likely to use one of the following approaches:

• Retry until successful

This approach can be applied transparently by the middleware layer provided that layer is able to deal with failover to an alternate instance as required by SV HA Config. This approach is simple to program if supported by the middleware.

This approach is not recommended, as there is significant potential to flood the network with retries during the failover or repair period when failures occur.

• Retry until a timeout is reached

This is similar to the previous model in causing flooding of the network with retries. This becomes almost indistinguishable from the option above if the timeout period is of long duration.

• Retry N times then fail

If N is sufficiently small, this avoids a continuous flood of retries, although the first one or two retries can be sufficient to cause flooding as described above. If this mechanism is used, the network must be sized to take retries into account.

• Do not retry but make local decision

In this situation, no retries are sent but the service controller must contain logic to determine if the service should continue without interaction with the Commerce Engine. This logic can be simple; for example, allow all service requests to proceed without authorisation if the expected value is below a threshold.

Protocol


Most middleware platforms and transport protocols support one or more of these options. It is important to be aware of the characteristics of the chosen platform, and the relevant issues are described in Transport Issues later in this chapter. In all cases, a mechanism is required to recover after failures when subsequent interactions depend on the previous behaviour, or previous behaviour requires some form of failure recovery. For example, if a service authorisation request is successful and the debit request fails, an appropriate recovery action must be taken to ensure that the debit request is delivered (for example, via batch processing of debit requests written to a file), or the reservation cancelled.

Transport Issues

The choice of transport protocol or middleware has significant implications for the protocol design, adapter design, and failover mechanism. This section provides an overview of the issues that should be considered when choosing a transport protocol or middleware platform. Because service authorisation and debit usage requests have different requirements, each is described separately where appropriate. There is some value in choosing separate transport mechanisms for each despite the additional complexity involved in supporting multiple transport mechanisms. In this section, 'transport mechanism' will be abbreviated to 'transport'.

One of the key requirements for service authorisation is low latency; the establishment of a call or service usage session should be nearly instantaneous from a user perspective. This is especially the case for highly-interactive, low-value transactions such as phone calls, but is also valuable for other services.

Usage debits are less sensitive to longer latencies because they typically occur after the service usage has completed.

Transports can vary considerably in their latency characteristics. Assuming the networking is consistent, the primary factors affecting transport latency are level of ordering and delivery guarantees provided. A reliable, guaranteed order transport such as TCP might cause delays on message delivery when IP packets are lost or delivered out of order.

UDP makes no guarantees and has very low latency but requires application level recovery mechanisms for lost packets. For Commerce Engine protocols, application-level recovery mechanisms are required and the ordering of requests is not significant, so the mechanisms applied by TCP are largely redundant.

Latency

Protocol


TIBCO offers a message-oriented protocol called TIBCO Rendezvous with minimal (configurable) delivery and ordering guarantees and relatively low latency. CORBA typically uses strong delivery guarantees via retries, but many implementations can be configured to provide an appropriate trade-off between latency and delivery guarantees.

Synchronous transport mechanisms can also significantly increase latency. For example, if a sender cannot proceed until a response or acknowledgment is received, unnecessary delays are imposed at the sending end. Many request/response mechanisms allow acknowledgments to be 'piggybacked' on the application level response, thereby avoiding this problem.

Transport delivery guarantees, while increasing latency, can significantly reduce the complexity of application code if used wisely. For service authorisations the latency is important and it would be reasonable to discard delivery guarantees in favour of low latency. For usage debits, however, delivery guarantees are potentially very useful.

TCP provides a very basic level of delivery guarantee using a framing protocol with piggybacked acknowledgments. It does not deal with failures gracefully, so is of limited use. TIBCO offers a guaranteed delivery protocol implementation over its TIBCO Rendezvous protocol, where it will transparently retry a request until delivery is successful or a (configurable) timeout occurs. It also allows application-defined 'delivery' so that the application can indicate when application failures result in non-delivery of the request. This is potentially very useful, particularly for usage debits to ensure that a request is stored on stable storage before confirming delivery. The cost of retries when problems occur must be taken into account (for more information, refer to Failure Handling Mechanisms earlier in this chapter).

Transactional messaging protocols such as MQ series offer even stronger delivery guarantees, including the ability to survive node failures yet still deliver requests through on-disk delivery queues. These could be useful for usage debits but the cost of retries must be factored into the transport decision.

The addressing capabilities of a transport are primarily useful in handling failover. The ability to use an address conveniently or transparently to access an alternate node in the case of failover is particularly desirable in a high availability environment. Multicast addressing can also be useful in delivering a request to multiple instances of a service to facilitate replication or load balancing.

UDP and TCP use IP addressing. For these transports, failover can be achieved by reassigning the IP address to another node. This is often supported by high-availability software and networking hardware. For TCP, the session context is lost and must be re-established. The relatively stateless nature of UDP avoids the need for re-establishing a connection.

Delivery Guarantees

Addressing

Protocol


CORBA uses object identifiers to address endpoints. Prior to CORBA 3 these identifiers were not easily transferred to an alternate node, so the ability to failover usually required application level support. CORBA 3 introduces support for fault tolerance and allows an object identifier to refer to a group of object instances spread across nodes.

TIBCO Rendezvous is a multicast protocol, supporting replication of services and also allowing transparent failover because the request is sent to all subscribed nodes.

Transactional messaging implementations typically allow explicit disk queue addressing from multiple sources, so failover is handled by having the alternate nodes read from the same request queue.

The ability of a transport to balance the delivery of requests across a set of adapter instances can be useful, as discussed in Chapter 3, Adapter Design. The load associated with requests can then be spread across multiple service instances.

UDP offers limited load balancing capability if the UDP socket is opened in a parent process and inherited by a set of children on the same node. The semantics of delivery depends on the host operating system, but is typically based on the order of read requests issued by the processes. TCP offers no similar capability, primarily because of the session state associated with ordering and delivery guarantees.

CORBA does not have a standard load balancing mechanism, although some implementations support load balancing through their object adapter implementations.

TIBCO allows load balancing through an alternate transport built above its TIBCO Rendezvous multicast protocol. The overheads of this additional protocol layer need to be weighed against the benefits.

Transactional message queuing mechanisms typically allow multiple processes to dequeue requests concurrently, providing transparent load balancing. The semantics of dequeue operation become important and should be examined to ensure that requests cannot be dequeued by more than one process, and that uncommitted dequeue operations do not prevent subsequent dequeues from occurring.

Load Balancing

Protocol


Example Protocol Design for Service Authorisations

Service authorisations are used to reserve account balances and ensure that customers do not exceed their credit limit during usage of a service. The service controller usually sends authorisation requests before the service usage is allowed to proceed (for example, during call connection for a phone call). They are potentially sent during the service usage to add to a reservation if the initial reservation is insufficient. In most environments, this usage implies that authorisation requests must have very low latency (tens of milliseconds or less) and that there will be a significant number of concurrent requests.

The financial risk associated with failure of a request depends on the application:

• In all cases, it is likely that the customer incurs an obligation to pay for the service usage, regardless of their balance and credit limit. This means that loss of a reservation only exposes the service provider to a credit risk if the customer is unable or refuses to pay.

• For high value transactions, the credit risk is clearly more significant.

• For prepaid customers, the likelihood of non-payment is higher as the agreement with the service provider is for a prepaid amount, and only minimal credit checks would have been applied.

• For post-paid customers, the likelihood of non-payment should be relatively low, as the customer typically has a time-based agreement with prior credit-worthiness assessment.

For this example, it is assumed that:

• A mobile voice service provider has high call volumes and a prepaid customer base.

• Individual call charges are of relatively low value, which means that a lost reservation has relatively low credit risk

• Any access controls associated with the service are dealt with separately (that is, provided by some mechanism independent of the Commerce Engine).

Given the information in Failure Handling Mechanisms earlier in this chapter, the decision for this example is not to retry but allow the call to proceed for an initial period and rely on a subsequent authorisation request or the service usage debit interaction. This ensures that balance is reserved or the usage is billed.

Protocol


In this case, all requests must carry all information necessary to perform service authorisation. This requires the service controller to hold this information, and the network bandwidth is slightly increased. This approach exposes the service provider to credit risk as outlined above, but as transaction values are typically low, it is acceptable. It avoids the overheads of processing retries and simplifies the protocol and requirements of the underlying middleware/transport. To help in dealing with failures, the service controller supplies a key with the service authorisation. This key is used for all interactions relating to that service provision.

For this example, it is assumed the TIBCO Rendezvous messaging protocol is used. It applies minimal delivery guarantees and is multicast, which supports the failover approach.

Having decided on the handling of errored requests by the service controller, the effect on the Commerce Engine must be considered. This is where the different modes of failure become significant. The modes of failure are described in the following subsections.

If the service authorisation request is not received or a Commerce Engine failure occurs before a reservation is created, a subsequent authorisation request or usage debit will contain sufficient information to establish a reservation or debit the account. Figure 2�2 displays this situation, showing the service controller reacting to a timeout by allowing the call to continue with a small number of initial units, then attempting the authorisation again once that initial allocation is used.

Service request

Authorise service

N units authorised

Service ready (initial K units)


Authorise service

N units authorised

CommerceEngine

Authorise service

Timeout

K unit timer

(N - K) unit timer

Figure 2�2 Service Authorisation Request Lost

Service Authorisation Request Lost

Protocol


If a reservation has been created but the reply is lost or the Commerce Engine fails after creating the reservation, the failure will be transparent providing the reservation is not lost in failover. Subsequent requests from the service controller will supply the same reservation key.

Figure 2�3 depicts this situation. The processing for the service controller is the same as for a lost service authorisation request, but the Commerce Engine must in this case recognise the resent request, and resend the appropriate response.

Service request

Authorise service

N units authorised (resend)

Service ready (initial K units)


Authorise service

N units authorised (resend)

CommerceEngine

Authorise service

Timeout

K unit timer

(N - K) unit timer

Authorise service

N units authorised

Figure 2�3 Service Authorisation Response Lost

If a timeout occurs during the processing of a reservation, the service controller approves an initial number of units and behaves in the same manner as the loss of an authorisation response illustrated in Figure 2�3. It should ignore the service authorisation response when it finally arrives. Alternatively, the additional authorisation request can be avoided by detecting the late arrival of the authorisation response and using it, but this is more difficult to implement on the service controller side.

If the service controller fails during the processing of the request, the service is not provided, and the service user will need to retry their service request. If an outstanding reservation remains as a result of the failure, a reservation timeout mechanism is implemented by the Commerce Engine to allow removal of the reservation after the timeout period has expired.

Service Authorisation Response Lost

Request Processing Exceeds Timeout

Service Controller Failure

Protocol


Figure 2�4 depicts this scenario. Note that the service controller will supply a different reservation identifier for the second authorisation request allowing the Commerce Engine to handle it separately from the first request.

Service request (eg. dial number again)Authorise service

N units authorised

Service ready (eg. call connected)


Authorise service

N units authorised

CommerceEngine

N unit timer

Service request (eg. dial number)Authorise service

N units authorised

Authorise service

N units authorised

Reservationtimer

Figure 2�4 Service Controller Failure

Note: The scenarios described in previous subsections Service Authorisation Request Lost, Service Authorisation Response Lost, and Request Processing Exceeds Timeout are indistinguishable from the perspective of the service controller. It will timeout and apply its failure handling mechanism (that is, allow the call to proceed).

A further consideration is the procedure to apply if a reservation is lost during failover due to the asynchronous replication protocol of the Commerce Engine. Using the described protocol, this can be handled transparently by the Commerce Engine because subsequent requests carry sufficient information to establish a new reservation or debit the account.

Failover of Commerce Engine

Protocol


Example Protocol Design for Usage Debits

Usage debits carry the information necessary to debit a customer account for service usage. The service controller sends usage debit requests to the Commerce Engine on completion of service usage, or to incrementally debit a customer account for longer-lived, high value transactions. Loss or failure of these requests directly affects the revenues of the service provider and must be avoided at all costs. This is significantly different from the financial risks associated with losing a service authorisation request. As the service controller is not usually delaying or denying service to the customer during usage debit requests, the latency of these interactions is of less significance than for service authorisation requests.

The following are considerations for this example:

• Assume a mobile voice service provider with high call volumes and a prepaid customer base.

• The debit request will not be retried when failure occurs.

• The service controller must write failed debit requests to stable storage and use subsequent batch processing mechanisms based on file transfer to process those requests.

• Any access controls associated with the service are dealt with separately (that is, provided by some mechanism independent of the Commerce Engine).

Most service controllers for mobile networks already have the capability to generate call data records and deliver them for batch processing at a later date.

From the service controller perspective, any request that times out must be considered a failed request. All such requests are later delivered to the Commerce Engine for batch processing. In terms of the failure modes described in Design Issues earlier in this chapter, the following subsections describe scenarios.

If the debit usage request was not received by the Commerce Engine, or was not committed before a Commerce Engine failure, the request is processed in the batch run with no duplicate detection required. This scenario is illustrated in Figure 2�5.

Debit Usage Request Lost

Protocol


Service usage complete (eg. hang up)

Debit for usage X

Service user Servicecontroller Adapter Commerce

Engine

Batch usage data, including X

timeout

Figure 2�5 Debit Usage Request Lost

If the Commerce Engine committed the debit usage request but the confirmation was lost, the request is processed in the batch run, now requiring duplicate detection. This scenario is illustrated in Figure 2�6.

Debit Usage Confirmation Lost

Protocol



Debit for usage X

Usage X debited

Service userService

controller Adapter

Debit for usage X

Usage X debited

CommerceEngine


timeout

Detectduplicate

Figure 2�6 Debit Usage Confirmation Lost

If the Commerce Engine committed the debit usage request but the confirmation was received too late, the request is processed in the batch run requiring duplicate detection. From a protocol perspective, this scenario is almost identical to that illustrated in Figure 2�6.

If the service controller fails during the request, it does not know whether the request was successfully committed or not. On recovery, it should assume that the request was not delivered and use the batch mechanism to deliver the debit request. The Commerce Engine again applies duplicate detection mechanisms to ensure the customer is not billed twice. The service controller must maintain sufficient information in stable storage to support this behaviour. This scenario is illustrated in Figure 2�7.

Debit Usage Confirmation Timed Out

Service Controller Failure

Protocol



Debit for usage X

Usage X debited


Debit for usage X

Usage X debited

CommerceEngine


Detectduplicate

Figure 2�7 Service Controller Failure

The Commerce Engine failure for usage debits is handled by ensuring that the usage debit request is written to stable storage before returning successfully to the service controller. Processing of the request proceeds asynchronously from this point to avoid long blocking times associated with the rating and charge generation process. The Commerce Engine guarantees not to lose any usage debits written to stable storage.

Failover of Commerce Engine

Protocol



Chapter 3 Adapter Design

The Commerce Engine provides service authorisation and account usage debits to service controllers in real-time. The volume and rate of requests is usually high and the Commerce Engine must be able to handle requests efficiently. The design of the adapter and its interaction with the TRE can have a significant impact on adapter performance and its ability to cope with the volume of requests received from the service controller.

The basic function of the adapter is to receive protocol messages, make TRE requests, and then collect the response and send an appropriate protocol message. This chapter describes the high-level design choices for the adapter and their performance and useability characteristics.

Commerce Engine Interface Functions

The adapter is used to access TRE functions for service authorisation and account usage debits. The name and parameters associated with those functions are not predefined and depend on the data payload associated with the interaction protocol and the specific functionality required by the Commerce Engine implementation. For maximum flexibility and reuse, adapters should support configuration of the function called in response to each received protocol message. In the case of an asynchronous adapter design, it is necessary to pass control information to define how and where the response to a request should be sent (for more information, refer to Asynchronous Design later in this chapter).

It is recommended that a simple function signature be supported to accept a hashtable for request data and minimal additional parameters for control information. This allows maximum flexibility in data payload, with any data-specific functionality being handled in EPM.

The adapter can then be deployed in any environment using the same transport and protocol, and the data payload can be modified as required to support additional requests or changes in payload during the system lifetime.

Adapter Design


For example, an existing TIBCO adapter can be configured to call any TRE function accepting two hashtable parameters:

• The first contains control information defining the TIBCO reply topic.

• The second contains request data.

Interaction Models

The TRE uses Tuxedo as the underlying middleware and has two basic interaction models:

• Asynchronous, where the calling client queues the request for the TRE service and continues processing without waiting for a response. Any response must be delivered through an alternate mechanism.

• Synchronous, where the calling client (the adapter) waits for a response from the called TRE service.

Adapters built using the synchronous model are simple, but as the TRE client development libraries are single-threaded, multiple instances of the adapter must be run to allow processing of requests in parallel. Parallel processing of requests is essential for the high volume, low latency requirements of real-time service authorisation. A load-balancing mechanism then becomes necessary.

Adapters built using the asynchronous model require two distinct adapters for:

• Accepting and queuing the incoming request

• Sending the response.

Load balancing is achieved through Tuxedo configuration in the asynchronous design. The asynchronous model is preferred unless the transport protocol or middleware chosen for interaction with the service controller has a load balancing mechanism. Flexible and robust load balancing mechanisms are difficult to write and maintain.

The two subsections Asynchronous Design and Synchronous Design describe the asynchronous and synchronous approaches in more detail.

For more information about the use of Tuxedo in Convergent Billing, refer to the Tuxedo Guide for Singl.eView Convergent Billing.

Adapter Design


Asynchronous Design

An asynchronous adapter design requires two adapter implementations:

• A TRE client that accepts protocol messages and calls an asynchronous Tuxedo function.

• A TRE service that accepts TRE function calls with parameters defining the response, and which sends a corresponding protocol message containing the response.

These are usually referred to as inbound and outbound adapters respectively. An inbound protocol request usually contains control information defining the transport or middleware address for sending the response. For example, a TIBCO message for which a response is expected will contain a reply 'topic'. This control information must be passed from the inbound to the outbound adapter via the function called in response to the message as discussed in Interaction Models earlier in this chapter.

The asynchronous design achieves scalability by balancing requests across a configurable number of Commerce Engine servers using internal Tuxedo queuing mechanisms. Responses can also be balanced automatically by Tuxedo across a configurable number of outbound adapter instances to minimise delays associated with synchronous processing. Replies to the service controller can potentially be delivered out of order so the service controller must match requests and replies. This design is illustrated in Figure 3�1.

Adapter Design


Service Controller

Inboundadapter

Commerce Engineservers

Queue

Outboundadapters

Tuxedo servers

Requests

Replies

Replies

Queue

Queue

Figure 3�1 Asynchronous Design

Any TRE function called by the inbound adapter should therefore have the following characteristics:

• The function should be defined as an asynchronous TRE function returning an integer value that is discarded by the inbound adapter.

• The function should accept any control parameters necessary for sending the response.

• Before exiting, the function should explicitly send any required response by calling an appropriate function on the outbound adapter, including any necessary control parameters.

The key advantages of asynchronous design are:

• Flexible and robust load balancing is implemented by Tuxedo.

• Throughput of a single adapter instance is high because it does not block. Laboratory tests indicate that per request processing times of less than a millisecond are possible in an inbound adapter, depending on the complexity of the data payload and wire data format.

Advantages of Asynchronous Design

Adapter Design


• Assuming a single adapter instance can provide sufficient throughput, all requests can be sent to a single transport address without using a front-end process, noting that context switches and inter-process communication tend to have a significant effect on latency.

• Response formats are generally more flexible because there are distinct parameter lists for requests and responses, and multiple response functions can be defined. For example, distinct functions on the outbound adapter might be used for a successful outcome and each error condition respectively.

Some disadvantages of an asynchronous design are:

• It is sometimes not possible to send a protocol response from a distinct process. For example, implementations of the RADIUS protocol often require that the response be sent from the UDP socket to which the request was directed. This is not possible with distinct inbound and outbound adapters.

• There are always at least three processes involved in the processing of a request:

− Inbound adapter

− Service or services performing request processing

− Outbound adapter.

• The single inbound instance is not scalable. If a single inbound adapter cannot handle the incoming request volume, multiple instances listening on distinct transport addresses are usually necessary with some level of load balancing pushed back onto the service controller or middleware infrastructure.

• Outbound adapters in particular can be difficult to develop, and require both an understanding of TRE server infrastructure and a TRE development environment.

Disadvantages of Asynchronous Design

Adapter Design


Synchronous Design

The synchronous design requires only a TRE client implementation but must include a mechanism for load balancing across multiple instances. The function or functions called by the adapter are defined normally and a single function signature defines both request and response. Distinct errors can be indicated through the use of appropriate TRE exceptions or through explicit construction of an error response in the receiving EPM functions

There is no standard way of load balancing across the synchronous adapter instances. Transport and middleware protocols often provide a default mechanism that can be used (for more information, refer to Chapter 2, Protocol). For example, UDP requests are distributed across multiple processes if the UDP socket is opened by a shared parent process and inherited by the children. Care is required because this mechanism does not necessarily behave consistently on all platforms.

The load balancing mechanism must also be manageable, allowing instances to be added and removed as required and implementing checks to ensure the continued operation of existing instances. If such a mechanism exists, a synchronous adapter can be simpler and more efficient than an asynchronous implementation. If a custom load balancing mechanism has to be implemented, it is preferable to use the asynchronous approach.

Figure 3�2 displays a synchronous design where load balancing is implemented either in the middleware, or as an explicit process defined by the adapter implementation. The 'load balancing' process is shown as a different type of entity because it might not be a separate process.

Adapter Design


Service Controller Adapter

Adapter

Adapter

Load balancing

Tuxedo servers


Figure 3�2 Synchronous Design with Explicit Load Balancing

Figure 3�3 displays a synchronous design where the service controller has internal load balancing logic, and can distribute requests across multiple synchronous connections to adapter instances.

Service Controller Adapter

Adapter

Adapter

Tuxedo servers


Figure 3�3 Synchronous Design with Service Controller Load Balancing

Adapter Design


The key advantages of a synchronous approach are:

• With robust and flexible load balancing, a synchronous solution is inherently scalable.

• If load balancing is provided by middleware, the solution is simple.

• The number of processes involved in request processing can be as low as two, provided the load balancing does not require a front-end process and each request is handled by a single TRE service.

• The TRE function implementations are simple and clean.

The disadvantages of a synchronous approach are:

• If middleware is not provided, load balancing is difficult to implement.

• Additional configuration expertise is required to deploy and maintain a solution (that is, load balancing is not part of Convergent Billing and must be configured and managed through an external mechanism).

• The management interface can be more difficult to implement (for more information, refer to Chapter 5, Management Interface).

Advantages of Synchronous Design

Disadvantages of Synchronous Design


Chapter 4 Failover

A highly available solution requires a mechanism to allow the service controller to near transparently continue operating against a backup or alternate copy of the Commerce Engine when a failure occurs. This chapter describes the alternate mechanisms that can be used and their relative strengths and weaknesses. The transport protocol used has a significant effect on this choice. For more information about transport protocols, refer to Chapter 2, Protocol.

In this chapter a hot backup approach to failover is assumed; that is, the Commerce Engine has primary and backup nodes. The backup node is operational, but not processing requests. In the event of a failover, requests are redirected to the backup node. The adapter must support this redirection, either through an appropriate transport choice, explicit mechanisms, or both. The adapter might not be running on the backup node. If adapter startup is relatively fast, the adapter can be started on failover rather than requiring an 'enable' operation on a running adapter.

IP Address Failover

IP address failover relies on the ability of the network or management software to transfer a nominated Commerce Engine IP address from the primary Commerce Engine node to another. After this transfer has occurred, IP packets sent by the service controller are automatically routed to the backup node.

The value of this approach depends on the protocol being used. For a UDP protocol, this works well because each packet is delivered independently of all others. If each request is contained in a single packet and the synchronisation of the primary and backup Commerce Engine instances is up to date, the backup instance should be able to pick up where the primary instance left off when a failover occurs.

Failover


TCP is more problematic because it is a stream-oriented protocol with frames and ordering constraints used to ensure correct delivery. If the stream is broken, the protocol loses the current stream state and cannot recover that state on the backup node. Layering of higher-level protocols over TCP inherits this limitation, but automated reconnection can hide the problem from the application. For example, the layering of HTTP over TCP can resolve this problem if each HTTP request is delivered by creating a new TCP session (optional in more recent HTTP revisions), but the performance overheads are likely to be significant.

The implementation of this mechanism is usually done by either a smart switch such as the Cisco LocalDirector or by cluster management software. The mechanism used will influence the required behaviour of adapters and the service controller:

• If a switch and transparent network address translation (NAT) is used, the migration does not require any action in the service controller or the adapters.

• If the redirection is achieved by transferring the MAC address, the adapter on the backup node will need to bind to the IP address after the transfer has occurred.

• If redirection is achieved by transferring the IP address from one MAC address to another, the adapter on the backup node will need to bind to the IP address after the transfer, and the failover process will need to include a notification to the service controller so that the service controller ARP tables are updated appropriately.

The performance and technical details of these alternate implementations depend on the chosen hardware or software. IP address migration can delay the failover process by a period of seconds depending on the mechanism in use. A delay of this length might be undesirable so this should be considered when choosing the mechanism.

Failover


Multicast

The use of a multicast transport allows the dynamic association of an adapter instance with a multicast address. The adapter 'binds' to the address as required and, when bound, receives all requests sent to that address. There are two ways this can be exploited to support failover between primary and backup adapter instances:

• Have both instances receive all requests, but only process requests in adapter instances when it is explicitly 'enabled' through a management interface.

• Bind to the multicast address only when instructed to do so through a management interface, and also unbind when instructed.

• Stop the adapter on the active node and start the adapter on the backup node.

Any option is reasonable and the choice will depend on the cost of delivering all messages to both instances, compared with the cost and delay associated with binding to an address on demand or stopping and starting adapters. Typically, the second option is most efficient. In the first two cases, operations to enable and disable processing in the adapter instance are required.

The instructions to enable and disable or stop and start adapters are configured into the failover interface of SV HA Config. If multiple instances of the adapter are running on a node for load-balancing purposes, it is preferable for the management operation to enable and disable those instances be sent once, and delivered to all.

Failover


Controller Managed

A controller-managed approach to failover between adapters relies on the ability of the service controller to explicitly switch from one adapter instance to another on failure. This is the default approach used by the RADIUS protocol, for example. The two distinct ways this approach can be implemented are:

• Failover is initiated through a controlled mechanism, and a message or instruction is sent to the service controller to re-route requests to the alternate node when the failover is complete.

• The service controller re-routes requests to the alternate node whenever it determines that a failure has occurred.

The latter approach is not recommended for several reasons:

• If multiple service controllers are used, each determines failure at different times and requests can be routed to both instances concurrently. Data synchronisation mechanisms do not support concurrent access.

• The likelihood of 'false' failovers is higher than for mechanisms that identify failures at the source.

• The failover needs to be communicated to the Commerce Engine prior to failover to ensure that any configuration changes necessary to support the service on the backup instances are performed.

The requirement that the controller-managed re-routing of requests be initiated by an external request is not always supported by service controllers. It might be necessary to disable this mechanism and use either of the approaches described in IP Address Failover or Multicast earlier in this chapter.


Chapter 5 Management Interface

The Commerce Engine is intended for deployment in a highly available environment. SV HA Config. relies on one or more redundant copies of the Commerce Engine spread across clustered hardware nodes. Any Commerce Engine adapter must also be capable of similar redundant deployment and include controls necessary to support SV HA Config. To facilitate failover and high availability operations, a high availability implementation requires mechanisms to achieve the following:

• Re-routing of requests from the service controller to the backup node on failover

• Ability to determine if an adapter instance is currently active and receiving requests.

• Ability to collect statistics about adapter request processing.

The management interface should be accessible through the TRE. If multiple instances are used with external load balancing, a mechanism must exist for the TRE caller to find and call each instance for statistics and status, or a front-end mechanism to provide aggregate statistics and status for all instances. A per-node aggregate interface is preferable but not mandatory.

The TRE libraries include a mechanism that allows a process to run an EPM interpreter and execute configurable EPM functions in response to management events received from the Commerce Engine. Where possible, this mechanism should be supported by an adapter.

Failover support in the adapter can be achieved either by stopping and starting adapters on the primary and backup nodes respectively, or providing 'enable' and 'disable' management operations for the adapter instances as discussed in the previous section. This choice will depend on project-specific requirements and the time required to start and stop adapters.

Management Interface


Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00 Index-1

Index

Adapters accessing TRE functions 3-1 asynchronous design 3-3 Commerce Engine 1-1 controller-managed failover 4-4 design 3-1 Design 1-2 design criteria 1-1 design issues 2-2 development guidelines 1-1 example protocol design 2-8, 2-13 external issues 2-1 failover support 5-1 IP address failover 4-1 load balancing 2-7 management interface 1-2, 5-1 multicast transport 4-3 network components and protocol design 2-1 prebuilt 1-1 protocol 1-1, 2-1 protocol design for failure scenarios 2-13, 2-

14, 2-15 protocol for failures 2-3 redirecting requests 4-1 service controller 2-3 start and stop, enable and disable 4-3 synchronous design 3-6 TIBCO 2-6, 2-7, 3-2 transport protocol 2-5

Commerce Engine account debit for service usage 2-2 adapters 1-1 authorising a service request 2-2 failover 2-12 Transaction Engine (TRE) interfaces 1-1

Commerce Engine Adapter See Adapters Conventions

used in this guide vi Document

conventions used vi how this guide is structured v related documents v

Failover 4-1 Commerce Engine 2-12

controller managed 4-4 delivery guarantees 2-6 failure modes 2-9, 2-11 generic failure handling approaches 2-4 IP address 4-1 primary and backup adapter instances 4-3 transport addressing capabilities 2-6

Management Interface adapters 5-1

Multicast Transport adapter instance 4-3

Related Documents v Structure of Document v SV HA Config.

failover interface 4-3 Tuxedo

asynchronous and synchronous 3-2 load balancing 3-2

Index

Index-2 Commerce Engine Adapter Developer's Guide for Singl.eView Convergent Billing v6.00