using ibm mq in managed file transfer environments final

© 2015 IBM Corporation

AME-2285: Using IBM MQ in Managed File Transfer EnvironmentsLeif Davidsen

Senior Product Manager

IBM Messaging

Files and file content are in every part of the enterprise

• Businesses and their applications use files across their

infrastructure to store valuable data

• Where the data is created is not always where it needs to be

consumed

• Files and file contents need to be moved to deliver value

1AME2285 - @LeifDavidsen

How do most organizations move files today?

• FTP• Typically File Transfer Protocol (FTP) is combined with writing

and maintaining homegrown code to address its limitations

• Why is FTP use so widespread?• FTP is widely available – Lowest common denominator

• Promises a quick fix – repent at leisure

• Simple concepts – low technical skills needed to get started

• FTP products seem “free”, simple, intuitive and ubiquitous

• Legacy File Transfer products• A combination of products often used to provide silo solutions

• Often based on proprietary versions of FTP protocol

• Can’t transport other forms of data besides files

• Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure – especially with SOA

• People• From IT Staff to Business staff and even Security Personnel

• Using a combination of email, fax, phone, mail, memory keys…

Most organizations rely on a mix of home-grown code,

several legacy products and different technologies … and

even people!


Shortcomings of Basic FTP

Limited Flexibility

Limited Security

Limited visibility and traceability

Limited Reliability

Unreliable delivery – Lacking

checkpoint restart – Files can

be lost

Transfers can terminate

without notification or any

record – corrupt or partial

files can be accidentally used

File data can be unusable

after transfer – lack of

Character Set conversion

Often usernames and

passwords are sent with file –

as plain text!

Privacy, authentication and

encryption often not be

available

Non-repudiation often lacking

Transfers cannot be monitored and managed

centrally or remotely

Logging capabilities may be limited and may only

record transfers between directly connected systems

Cannot track the entire journey of files – not just from

one machine to the next but from the start of its

journey to its final destination

Changes to file transfers often require updates to many

ftp scripts that are typically scattered across machines

and require platform-specific skills to alter

All resources usually have to be available concurrently

Often only one ftp transfer can run at a time

Typically transfers cannot be prioritized


FTP introduces risks in data movement

Process Risk

Delays in transferring files impacts collaboration with customers/partners

Integration files that are delayed impact SLAs

Failure of file delivery impacts the processes themselves

Security Risk

Data encryption and governance of sensitive information transmitted in files

Inability to apply corporate security policies to person-initiated file transfers

No visibility over the type and sensitivity of the data being transferred

No ability to support audit requirements

IBM’s Managed File Transfer solution addresses all of these risks


Who benefits from Managed File Transfer

• Application Owners• Producers and consumers of data held in files

• Application developers likely to own files, own code and own existing file transfer mechanisms

• Increase reliability, improved timely access to data

• Infrastructure owners• Currently provide connectivity between systems, through middleware solutions

• Provide technical expertise to assist application owners

• Reduce recurring error handling and recovery, leverage middleware tools

• CxOs• Need to improve productivity of staff and IT

• Speed up business performance without impacting security

• Customers• Less risk of loss or exposure of data

• Improved business effectiveness and better, more accurate fulfilment


IBM MFT Vision

Accelerate and simplify governance of the growing volume of business-

critical data movement within and beyond the enterprise with Smarter MFT

Maximize the agility and performance of dynamic business networks by reducing

the complexity, risk, and cost of file transfer

Visibility• Single view of transfer activity

• Transaction and business monitoring• Dashboards, analytics, and scorecards

Usability & Management• Persona-based, easy-to-use interfaces

• Unified control and configuration of infrastructure

• Community on-boarding and coordination

Universal• Any transport, any protocol, and any

partner• Global and cloud-enabled deployment• Broad platform coverage and industry

standards

Security & Performance• Assured delivery and high availability

• Protection of file data in transit and at rest• Accelerated transport and low latency

Connectivity• Integration with BPM to drive business processes

• Leveraging ESBs to enable service orientation• SOA Registry/Repository for lifecycle governance


Comprehensive IBM Managed File Transfer

Monitor file transfer activity & SLAsIBM® Sterling Control Center

Peer-to-peer file transfers via file

systemIBM® Sterling Connect:Direct®

Message Centric

TransfersMQ MFT

Scope of this presentation

• A single packaged offering for IBM Managed File Transfer

• Sterling Connect:Direct and Sterling Control Center are packaged with

IBM MQ Managed File Transfer as WebSphere Managed File Transfer

• IBM MQ Advanced also includes these additional capabilities

B2B Onboarding and file transfer via

mailboxingIBM® Sterling File Gateway

Cloud service for B2B file exchangeIBM® Sterling File Transfer Service


IBM offers comprehensive MFT Capabilities

IBM MQ Managed File Transfer provides file

transfer optimized for data delivery across IBM MQ

networks

Sterling Connect Direct provides peer-to-peer file

transfer optimized for data delivery within and

between enterprises across Connect:Direct

protocol

Sterling Control Center Manage file transfer

activity across your file transfer servers including

Connect:Direct, FTP, MQ MFT and Sterling File

Gateway

Addressing multiple use cases and scenarios for both internal and multi-

enterprise file transfer


MQ V8 Advanced – Universal Messaging including Managed File Transfer

WebSphere Managed

File Transfer

All functions available for one single price

Sterling C:D

Sterling C:C

MQ V8

MQ Telemetry

MQ AMS

MQ MFT

Sterling C:D

Sterling C:C

MQ MFT

MQ Advanced V8 Enterprise Messaging with publish/subscribe and

queuing for any application, on any OS, using

any language

Connects applications reliably and securely to

send data anywhere to any solution

Extends application messaging to Mobile,

Internet of Things and Managed File Transfer

Single, simplified licensing to provide all the

capabilities to connect the business, and

maximize data value

Do more with your MQ investment, lowering

operational overheads, and raising efficiencies

Ensures all possible functions and capabilities

are available instantly to allow the business to

respond faster to changing needs

Moves data from anywhere to anywhere with

included end to end monitoring and management

MQ Advanced V8

Available as WebSphere Managed File

Transfer Service, or WebSphere Managed

File Transfer Managed Endpoints


A consolidated transport for both files and messages

• Traditional approaches to file transfer result in parallel infrastructures

• One for files – typically built on FTP

• One for application messaging – based on IBM MQ, or similar

• High degree of duplication in creating and maintaining the two infrastructures

• MQ MFT reuses the MQ network for managed file transfer and yields

• Operational savings and simplification

• Reduced administration effort

• Reduced skills requirements and maintenance

File TransfersApplication

Messaging

Consolidated Transport

for Messages & Files


Applications exchanging file data

Components of a typical MQ MFT network

Agents

– The endpoints for

managed file transfer

operations

Commands

– Send instructions to agents

Log database or file

– A historical record of file

transfers

Coordination queue manager

– Gathers together file

transfer events

IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands Log databaseor file


Log databaseor file

Agents

Act as the end points for file transfers

Long running MQ applications that transfer files by splitting them into MQ messages

– Efficient transfer protocol avoids excessive use of MQ log space or messages building up on queues

Multi-threaded file transfers

– Can both send and receive multiple files at the same time

Generate a log of file transfer activities which is sent to the “coordination queue manager”

– This can be used for audit purposes

Associated with one particular queue manager (or MQ Appliance)

Agent state on queues


IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands


Log databaseor file

Commands

Send instructions to agents

and display information about

agent configuration

– Via MQ messages

Many implementations of

commands:

– MQ Explorer plug-in

– Command line programs

– Open scripting language

– JCL

– Documented interface to

program to


IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands


Log Database & File

Keeps a historical account of

transfers that have taken place

– Who, where, when… etc.

Implemented by the ‘logger’

component which connects to the

coordination queue manager

– Stand alone application

- Can log to database or file

– Or JEE application

- Can log to database only

Queryable via Web Gateway

– Also a documented interface


IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands Log databaseor file


Log databaseor file

Coordination Queue Manager

Gathers together information

about events in the file transfer

network

Not a single point of failure

– Can be made highly available

– Messages stored + forwarded

MQ publish / subscribe

– Allows multiple log databases,

command installs

– Documented interface


IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands


Log databaseor file

MQ MFT and the MQ Appliance

The MQ Appliance can be the Queue

Manager to provide both regular MQ

Queue Manager capabilities

No other MQ deployment needed

Also Coordination Queue Manager

capabilities

No files are stored on the appliance

No MQ MFT Agent needed on

appliance to support this

Highly available and robust

Secure with MQ AMS entitlement

built in

– Content encrypted based on policies


IBM MQ

Agent Agent Agent

“Coordination”

Queue Manager

Commands


Example usage of monitoring + program execution

1. Application writes

file to file

system

Existing

Application

MQ

MFT

Agent

MQ

MFT

Agent

Existing

Application

2. Agent monitors file

system, spots arrival

of file and based on

rules, transfers the file

3. MFT transports file

to destination

4. At destination MQ MFT

writes file to file system

5. MFT can also start another

application to process the file


Securing file data with SSL and IBM MQ AMS

MQ MFT supports transport

level encryption using SSL

Data is encrypted before it is

sent over a channel and

decrypted when it is received

IBMMQ

QueueManager

IBMMQ

QueueManager

svrconn channel

sndr/rcvrchannels

IBMMQ

QueueManager

IBMMQ

QueueManager

svrconn channel

sndr/rcvrchannels

When combined with MQ

Advanced Message Security

– Allows file data to be encrypted

at the source system and only

decrypted when it reaches the

destination system

– Data is secure even when at

rest on a queue

Agent Agent

AgentAgent

AME2285 - @LeifDavidsen

Handling both files and messages across MQ MFT

One file to one message

MQMFT

One file to a group of messages

One message to one file

A group of messages (or all messages on the queue) to one file

• File can be split based on:

• Size

• Binary delimiter

• Regular expression

• One message becomes one file

• Optionally, a delimiter can be

inserted between each message

used to compose the file

• One file becomes one message

MQMFT

MQMFT

MQMFT


Universalmessaging backbone

IBM MQ Advanced V8 Architecture Summary

WebSphere Managed File Transfer (Includes MQ MFT, Sterling C:D, Sterling C:C)

• End-to-end audit trail across file transfers

• Reliable and efficient transfer of file data regardless of size

• Centralized monitoring of transfer events and management of transfers

MQ Advanced Message Security

Signs and encrypts data sent over MQ

to provide privacy and integrity

IBM MQ

Scalable, reliable, and efficient transport

for data

File-to-File

Message-to-File

File-to-Message

Message LevelSecurity

Mobile & M2M Messaging

IBM MQ Telemetry

Lightweight, low power, low bandwidth

messaging to mobiles and sensors


A grocery retailerIBM messaging backbone delivers transaction data to central hub in near-real time

The need:

At this company, transaction data from more than 2,400 grocery stores

reached a central data warehouse via daily batch transfers, delaying

analysis and hampering theft prevention efforts.

The solution:

The company implemented an integrated message queuing

infrastructure that can transfer data from any store endpoint to the

central hub in near-real time

IBM MQ MFT software transfers files between the central hub and the

stores’ ESBs and in turn to other endpoint applications in the store.

The benefit:

Dramatically reduces time between customer transaction in

store and data reaching data warehouse

Provides a rock-solid messaging integration engine in a small footprint

Delivers a reliable infrastructure for transaction data and

file-based information

“WebSphere Message

Broker is rock-solid. It has

a relatively small footprint,

and it is a world-class

integration engine.”

—A software engineer,

grocery retailer

Solution components:

Software

IBM® Integration Bus

(formerly known as

IBM WebSphere®

Message Broker)

IBM MQ

IBM MQ Managed File

Transfer





networks




protocol




Gateway

Addressing multiple use cases and scenarios for both internal and multi-enterprise file transfer


IBM® Sterling Connect:Direct

• What it does– High volume, large size, mission-critical file transfers

– Automation, retry-restart, for unattended operation, with never breached security

• What it allows clients to do– Move files with confidence and upgrade unreliable, unsecured FTP

– Handle growth in data volume and size without increase in staff

• How do I know I need it?– Regulatory, industry, or company security/compliance requirements continue to grow

– Business operations require assured delivery of critical data

Reliable and high performance solution for secure, point-to-point, bulk data transfers

Peer-to-peer file transfersIBM® Sterling Connect:Direct®


Detail: IBM® Sterling Connect:Direct

• MFT solution using peer-to-peer file transfer to move data across the

enterprise and between trading partners

• Remediates FTP and failed audits by adding security, monitoring, and

reliability without replacing FTP scripts

• High-volume, bulk data transfers for corporate-to-bank and financial

data connectivity

• Data movement across a broad range of platforms, including

mainframe, midrange, and distributed

• Small footprint, robust integration with 3rd party schedulers, and rapid

deployment


Trading Partner

Integrating IBM Sterling Connect:Direct and IBM MQ MFT

MFT

Audit

Agent

IBM MQ

Agent Agent

C:D

Bridge

Agent

C:D

Node

C:D

Node

C:D

Node

C:D

Node

C:D

Audit

Reference

Inside the MFT audit trail…

The audit information for each MFT transfer

references related C:D audit information

The Connect:Direct Bridge

capability supports managed file

transfers that span MFT and

C:D with a joined up audit trail


Bridging MQ and C:D networks at a large publisher

Solution architecture for a large publisher using the C:D bridge as a way to consolidate its IT infrastructure while still retaining the communication protocols used to communicate with

partners

MQ

Pain Points

Existing solution is a hodgepodge of

different data transports from different

vendors

Difficult to integrate existing applications,

built on MQ messaging, with file data

arriving from business partners

IBM’s MFT suite helps

Bridge the MQ and C:D environments

Leverage existing MQ skills and investment

Message-to-file and file-to-message

End-to-end visibility of files and messages

Modernize batch applications

Modernize MFT with RESTful Web 2.0

interfaces

Internal connectivity standardized on an MQ-backbone

Existing MQ applicationsCustomer #1

Customer #2

Customer #n

…

C:D bridge

Customers request that publications

can be submitted to the publisher via the C:D

protocol

Publisher

28




networks




protocol




Gateway



IBM® Sterling Control Center

Delivering higher levels of service for key integration and transfer services

• What it does

• Consolidated logging, reporting, and analytics for all transfers and integration

processes

• Rule-driven service level management

• Alerts and notifications

• Configuration management

• How it is delivered

• Java-based solution for management and visibility of the file transfer and B2B

integration infrastructure

• Engine, agents, and user consoles

• Interfaces to existing system management tools


IBM® Sterling Control Center

• What it allows customers to do

• Have visibility into all file transfer activity including Sterling Connect:Direct and

IBM MQ MFT from a central console

• React to transfer events before they impact SLAs

• Easily provide compliance reporting and mobile monitoring

• Central configuration management for Sterling Connect:Direct® servers

• How do I know I need it?

• Increasing numbers of transfers or B2B interactions are governed by SLAs

• Increasing compliance, governance and audit requirements

• Pressure to decrease IT operational cost for B2B or file transfer operations

• What differentiates IBM Sterling?

• Product specific knowledge of file transfer events, business processes and SLAs

to build rules around

• Access from mobile devices

• Scale and performance – test environment with moderately-sized hardware

performed well with 5K servers producing 540K processes and 4.3M events

per day





networks.




protocol




Gateway


Available together as WebSphere Managed File Transfer, or as Sterling

Connect:Direct Advanced. These entitlements are also combined inside the

IBM MQ Advanced offering


Adding MQ Managed File Transfer to your infrastructure

• Key benefits and highlights

• Add reliability, manageability, security when moving your file data

• Infrastructure team can manage a wider set of data movement through a single middleware layer

• Speeds delivery of value to businesses from file-based assets

– File to message; message to file

– File to file transfer also available

– Also bridge to FTP

• Deploy flexibly throughout your infrastructure to match your MQ environment

• Deploy MQ MFT agents where your files are

• Deploy MQ Advanced or MQ MFT Service on your MQ Queue Managers

• Leverage the MQ Appliance to provide MQ services in a MFT solution


1990s 2000s 2010s

V2 V5 V5.1 V5.3V5.2 V6 V7 V7.0.1V1.1

Multi-platform Managed File Transfer

V7.1 V7.5

System PatternApplication Pattern

MQIAssured delivery

Integrated Messaging

Hypervisors

JMS 1.1RCMS

Common criteriaEclipse

HTTPAJAXREST

Web 2.0

.NET

SSLSOAP

JMS

XML

Mobile

V2.1

Pub/sub

Multiple clusterXMIT queue

Multi-instanceQMGR

20th Anniversary of MQ at IMPACT

Timeline for MQ and Messaging product releases

V8

IBM MQ

Advanced for

Developers

IBM MQ

Advanced

IBM

MessageSight

IBM MQ

Advanced

Message

Security

IBM MQ

Managed File

Transfer

IBM MQ Low

Latency

IBM

MQSeries IBM MQTTIBM MQ

Everyplace IBM MQ

Appliance


Notices and Disclaimers

Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or

transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with

IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been

reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM

shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,

EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF

THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT

OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the

agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without

notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are

presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual

performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,

programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not

necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither

intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal

counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s

business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or

represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published

announcements or other publicly available sources. IBM has not tested those products in connection with this

publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM

products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to

interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,

INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A

PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any

IBM patents, copyrights, trademarks or other intellectual property right.

• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document

Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,

ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,

PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,

pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,

urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of

International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and

service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on

the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml

Thank YouYour Feedback is

Important!

Access the InterConnect 2015

Conference CONNECT Attendee

Portal to complete your session

surveys from your smartphone,

laptop or conference kiosk.