vocalink faster payment service architecture

© 2010 Sombers Associates, Inc., and W. H. Highleyman / Damian Ward (VocaLink)www.availabilitydigest.com

For discussion, contact [email protected]

www.availabilitydigest.com

Faster Payments – Bringing Payment Processing Into the 21st CenturyJune 2010

Damian Ward, VocaLink

Abstract

Faster Payments is the first new payments service to be introduced in the UK formore than twenty years. For the very first time, it has enabled phone, Internet andstanding-order payments to move in near real time - almost at the touch of a button.

On 27 May, 2008, VocaLink implemented the hub of the Faster Payments Service(FPS) in the UK. The Faster Payments Service is revolutionary and is the first application of the VocaLinkReal-Time Payments Platform.

The Faster Payments Service operates on a 24x7 basis, providing near real-time interbank transfers forInternet and telephone-banking payments. The delivery of this unique service has changed the dynamics ofconsumer banking behaviour and the payments industry.

This paper presents a brief overview of the development, implementation and operation of the UK FasterPayments Service and why it is able to operate at 100% service availability thorough the use of active/activetechnology.

http://www.availabilitydigest.com/



Award Winning

VocaLink triumphed at the Tenth Anniversary Financial Sector Technology (FST) Awards, winning the ‘bestof the best’ trophy for the Faster Payments Service.

In 2010, the FST Awards celebrated their tenth anniversary; and to honour theoccasion they broke open the vaults and looked back over previous winning entriesto select what they considered to be the ‘best of the best’ out of all the projectsrecognized over the last decade.

The judges were looking for noteworthy, revolutionary projects that have changedthe industry. At the awards ceremony held at the London Lancaster Hotel, over 500attendees from across the industry saw VocaLink’s Chief Operating Officer, IanGausden, collect the winning trophy on behalf of VocaLink, chosen as the overallwinner for the Faster Payments Service.

About VocaLink

VocaLink is a specialist provider of transaction services to banks, their corporate customers, and governmentdepartments. On a peak day, the VocaLink automated payment platform processes over 90 milliontransactions. Annually, over 9 billion transactions are processed:

Its switching platform connects the world’s busiest ATM network of over 63,000 ATMs. Its Real-Time Payments platform provides the central infrastructure for the UK Faster Payments

Service.

VocaLink are also working with BGC (Bankgirocentralen) to process most of Sweden’s domestic payments.

With Voca’s heritage of bulk payment processing and LINK’s understanding of real-time transactionswitching, both companies were already recognised as leaders in their respective fields. As pioneers ofelectronic transactions, the companies saw each other as natural partners to deliver this vital service to thepayments industry. A strong working partnership was forged. On 2 July, 2007, Voca and LINK merged; andVocaLink was born.

VocaLink has a proven history of successfully delivering new and innovative services in the highlycompetitive payments arena.

Faster Payments Background

The UK Faster Payments initiativehas its origins in the CruickshankReport, which was published in 2000.The report highlighted the need for alow-cost way to transfer moneyquickly.

In May, 2005, the Payment SystemsTask Force, chaired by the UK Officeof Fair Trading, announced that anagreement had been reached withthe UK banking industry to reduceclearing times on phone, Internet andstanding-order payments. The UKbanking industry committed todevelop a system whereby paymentswould clear in half a day.

Voca and Link made the conscious decision to look at the concept of faster payments imaginatively. Ratherthan settle for incremental improvements in clearing times, they looked to design, build and implement aReal-Time Payments Platform that would offer strategic benefits to the payments industry.

http://hm-treasury.gov.uk/media/654/71/2YRCshank-251104.pdf

http://hm-treasury.gov.uk/media/654/71/2YRCshank-251104.pdf



In October, 2005, a joint proposal from Voca and Link was selected to deliver the payment-processinginfrastructure for the Faster Payments Service.

When the Faster Payments Service went live in May, 2008, it far exceeded the original requirements. It wasmore ambitious, more innovative and more exciting for users than the original proposition. Instead of sameday or next day clearing, payments appear in the recipients’ bank accounts within seconds of being initiated.

Implementation Time Scales

The success of the Faster Payments programme depended on having the right technology infrastructure inplace. In building the infrastructure, VocaLink faced many challenges, including a fixed (and public) go-livedate and a fixed budget.

VocaLink, in partnership with APACS (Association for Payment Clearing Services, the U.K. paymentsassociation) and 13 participating banks, were confident in their ability to deliver a high-quality service to theUK market. The Real-Time Payments Platform design was performed in-house at VocaLink working with keysuppliers.

The real-time transaction switch uses a customised version of the FIS (formally eFunds) Connex Advantageapplication running on HP NonStop server hardware. This is coupled to a bespoke

1back-office and

settlement engine built using the Java/Oracle/Sun technology stack. The end result is a robust and fastsystem, sized to be able to cope with the projected high volumes of Faster Payments traffic for years tocome.

The decision by VocaLink to provide real-time technology offers the greatest business benefits to banks,consumers, and corporate customers. The VocaLink strategy reflects a belief that there will be continuedconvergence between customer channels, such as Internet, mobile, point-of-sale and ATM. The bankingindustry will need to support these channels, and to do so individually is time-consuming and expensive.

Faster Payments Launch

The Faster Payment Service was launched on 27 May, 2008. Payment volumes during the first daysurpassed industry expectation by 300%. Within the first three weeks, 1.7 million payments with a value of£1 billion had been processed. By the end of 2008, around two-thirds of UK phone and Internet paymentswere processed through Faster Payments.

The Faster Payments Service has delivered a world-class solution. It is operating under ever-increasingvolumes and is performing beyond expectations. In major system implementations, there are usually teethingproblems; but this service launched with no significant issues and has been recognised by the bankingindustry and HM Treasury as an outstanding achievement.

Volume Requirements

Sized to process 300 million payments annually: Equates to 542 tps at each site at launch Growing to 717 tps in 2011

1 For our U.S. English readers, “bespoke” means “custom.”



Availability Requirements

24/7/365 Service Availability SLA: 100% during the 00:00 – 06:00 Standing Order batch

run SLA: 99.75% overall Service credits and penalties apply for exceeding or failing to

meet SLA.

Faster Payments Technology

In the design of the Faster Payments System, VocaLink took a holisticview of availability. The service availability is supported through:

Application deployment model (dual site operation, eachcapable of 100% of workload).

HP NonStop server hardware and NonStop operating system. Connex Advantage Application design (NonStop

fundamentals). Faster Payments Message Protocol (designed to always fail

safe, retries permitted). VocaLink high-availability network infrastructure (dual

connections to all data centres). VocaLink data centre locations (geographic separation). VocaLink power and environmental systems (redundant,

backed-up power and cooling).

The Faster Payments Application

The FPS Core switch is built from the Connex Advantage full-functionEFT switch baseline:

Same baseline as used for the LINK ATM switch. VocaLinkhave almost 20-years experience with this platform or itspredecessors.

Leverages HP NonStop fundamentals, including NonStop process-pair technology. High performance core processing engine. Pathway subsystems for services around the core. Load-shared active/active single database configuration. Near-linear scalability in line with NonStop hardware expansion. Throttles workload to protect self and scheme participants.

Dual HP NonStop Servers Per Site

At the time of deployment, the highestperformance HP NonStop platformavailable was the NS16200. In order tomeet the volume requirementsstipulated by the scheme, 16 CPUs persystem were required. However, thiswould have resulted in no rapidupgrade path should volume be higherthan predicted. The decision was madethat each site would have twoNS16200 HP NonStop servers witheight CPUs apiece.

The application was split across thesetwo servers, with the core applicationon what is known as the ‘switch’ serverand pathway services placed on the‘services’ server. The switch and



services servers are clustered using Expand over ServerNet, and the result is near-equal workload on eachserver.

Performance testing of the distributed application showed negligible performance degradation after makingthis change. Measurements indicated that interprocess communication times across the Expand networkwere only slightly higher than normal CPU to CPU interprocess communication timings.

VocaLink now have the ability to add additional CPU resources to each HP NonStop server withoutsignificant effort should this be required. Each site with its two clustered servers forms a Faster Paymentsprocessing node.

Service Availability

The FPS scheme stipulated a veryhigh SLA for the FPS service, andfrom day one a dual-site strategy waspursued. However, the existing dual-site processing model used by theConnex Advantage application did notprovide the features required byVocaLink for the FPS service.

In the standard Connex Advantagemodel, each external memberconnects to either node A or node Bbut significantly not to both.Transactions are routed betweenprocessing nodes to find theirdestinations as required. Members willstill see a small outage each time theyare switched from one FPS node to the other following a failure or during maintenance.

Additionally, in a distributed processing model with an even number of members on each node, significantlevels of transactions need to be processed on both nodes. This will occur where the sending and receivingmembers are connected to different FPS nodes. This mode of operation is known as CP (or ContinuousProcessing).

VocaLink Active/Active Model

In the VocaLink active/active modelof operation, scheme membersconnect to both FPS processingnodes simultaneously. Membersalternately send transactions to eachFPS processing node.

When one node fails or is takenoffline for maintenance, any sessionsto it are closed down; and memberprocessing continues uninterruptedusing the remaining node.

When the offline FPS processingnode comes back into service, itssessions are reestablished with themember systems; and processingresumes.



In the event of transient errors, transactions are still able to cross from one processing node to the other inorder to find an active path to their destination. Live running shows that this scenario is only followed for avery small number of transactions. The vast majority are processed entirely on one or the other processingnode.

This processing model allows a single node to be taken out of service for maintenance without loss ofservice to the members.

Since launch the FPS service has maintained a service availability of 100% despite scheduled single-siterunning to cater to application updates or as a precautionary measure while power or other environmentalwork is undertaken. In the last year, the service has switched four times to a single-site configuration to caterto this upgrade work.

Protocol Fail Safe

As part of the overall FPS system design, the application protocol was designed to always fail safe (i.e. thestate of a transaction is always known).

The FPS Switch performs duplicatemonitoring to prevent the sametransaction being submitted morethan once in error. As part of thefailsafe protocol scheme, membersare permitted to resend transactionsonce they time out. When theoutcome of the original or resenttransaction is known by the FPSSwitch, it will send the completedtransaction outcome to the submitter.When the transaction outcome is notknown by the FPS Switch, thetransaction is resubmitted to the receiver.

Other failsafe features of the protocol include reversal on timeout and the processing of payments in asynchronous and asynchronous manner.

SIP transactions are processed in a synchronous mode, where the transaction must be accepted by thereceiver before an acknowledgement is sent to the submitter. SO payment transactions are processed inasynchronous mode, allowing the FPS Central Infrastructure to acknowledge the submitter prior to sendingthe payment to the recipient.

FPS supports 98 payment flow and recovery scenarios.

Ensuring Quality of Service

FPS supports three major transaction types:

SIP (Single Immediate Payment) - transactions where a response is required by the end user within15 seconds.

SO (Standing Order) - payments where a batch needs to be processed within a set time frame, butindividual transaction timings are less important.

Unsolicited Messages (USM) - inform participants of the states of the FPS Central Infrastructure andother member systems. These messages are used to communicate management informationbetween participants.

It is essential to maintain adequate response times for SIP transactions at all times, as these have end-userswaiting for the transactions’ outcome responses. To ensure this is possible, each transaction class usesdedicated communications sessions for both the sending and receipt of payment transactions.



Cross-Site Replication

In order to support the VocaLinkactive/active processing model,extensive use is made of replicationproducts, specifically the GoldenGatereplication engine.

Files are either synchronised acrosssites to maintain consistent systemcontext or are copied to a shadowlocation on the remote node shouldrecovery be required. Latency-criticalfiles such as the encryption keys arereplicated by the Connex Advantageapplication itself.

Delays in replication of data can cause failures in processing transactions. The FPS Central Infrastructuremitigates the risk in three ways:

Ensures that data is replicated quickly. Uses logic within the application to safeguard against delays. An application design that ensures that data replication fails safe.

How to Control Volumes

The FPS scheme membership comprises members of varying sizes with different transaction volumerequirements. In order to ensure the FPS Central Infrastructure and transaction receivers are not floodedwith Standing Order payments, a simple but effective method of throttling is applied. Each member has apreset number of sessions configured between it and the FPS service. Each session can have only a singletransaction outstanding, and a session cannot be reused until a response is received or timed out for thatsession.

This method of control, although simple, shows how smart system design can be used instead of a morebrute-force approach. The method has been proven to protect the FPS system and its participants andmaintain SIP performance even during the SO batch run.

Faster Payment Settlement System

The FPS Settlement System supports configurable settlement cycles, normally three per banking day at15:45, 07:00 and 13:00.

The FPS Settlement System alsoprovides transaction and settlementposition enquires to scheme members.The following ad-hoc settlement queriesare supported:

Multilateral Net SettlementPosition

Bilateral Settlement Positions

The Multilateral Net Settlement Positionshows all of the members’ positionsagainst each other.

Bilateral Settlement Positions show an individual member’s position against each of the others. A membercan only enquire on their own settlement positions.



Fraud / Money Laundering Risks

The entire UK Payment Industry was wary of the implications of the FPS scheme. The idea of money flowingaround so quickly and its potential for abuse caused concern with regard to Fraud and Money Launderingspecifically.

In order to address these concerns and mitigate the risks, many settlement, risk, and limit checks have beenbuilt into the FPS system, including single transaction limits, maximum member liability limits, and liabilitywarning thresholds.

There is a maximum system-wide value set for any single transaction - no item processed by FPS canexceed this limit.

Summary

The FPS system is a success story of which VocaLink are rightly proud. It has brought about a step changeover traditional payment channels and has positioned the UK payments infrastructure well for the removal ofpaper cheques in 2018.

The ‘end-to-end active/active’ model works to maintain service availability. This is proven each timeVocaLink take a processing node in and out of service for maintenance without business-service impact.Specifically, having scheme members connected simultaneously to both FPS processing nodes is essentialin ensuring system availability.

The ease in which a processing node may be removed and reintegrated with the FPS system has madeVocaLink more confident with regard to change management and change scheduling. Riskier changes maybe undertaken because the FPS service can easily be isolated at one processing site while maintenance isperformed at the other. Application versions and even new NonStop hardware can be added withoutimpacting the service availability in any way.

FPS scheme volumes are increasing - the announcement by the UK Payments Council that paper chequeswill be phased out in the UK by 2018, displacing four billion payments per annum into other channels, willincrease volumes significantly.

VocaLink are confident the model will scale as they have only 8 CPUs per system at present. Using thecurrent architecture (NS16200), VocaLink can double the compute power of the existing infrastructure. Amove to HP NonStop Blades (NB50000) will allow each FPS site to be reduced to a single-system footprint.Introduction of quad-core CPUs will reduce the footprint further and will allow for massive growth.

There are many different ways to replicate data; and each replication technique must be analysed, takinginto account the sensitivity to latency and the probability of collisions when selecting the most appropriatereplication method.

With thought, some aspects of the system can be designed to be failsafe in the event of a data replicationfailure or delay. Where this is not possible, real-time data is best replicated by the application.

The Faster Payments Service has revolutionized payment settlement, moving it from a slow batch process toa nearly instantaneous, interactive service. FPS’ use of active/active technology to eliminate planneddowntime and to survive unplanned failures ensures the continuous availability of settlement services to theretail community.

vocalink faster payment service architecture

Documents