ibm connectivity and integration

© 2013 IBM Corporation

IBM Connectivity and Integration

Lee Gavin

European WebSphere Technical [email protected]

22 © 2013 IBM Corporation

Messaging backbone in the data center

MQ

The World As We Knew It

Application platforms provide containers for application and buisness logics

WAS, CICS, IMS



MQ

The Changing World: Towards SOA

Integration Bus

Enterprise Service Bus integrates apps/data/services and partners, appliance provides integration gateway to heritage services


WAS, CICS, IMS


Cache grids improve scale and performance of applications and services


MQ

Secure appliances enable controlled access to Enterprise resources

DataPower XG45

WXS, DataPower XC10

The Changing World: Emergence of Appliances

Integration Bus



WAS, CICS, IMS




MQ


DataPower XG45

WXS, DataPower XC10

Public Cloud

The Changing World: Budget Versus Architecture

Connectivity to applications in the public cloud enables Enterprises to leverage a new cloud economy

Cast Iron, DataPower XH40

Integration Bus



WAS, CICS, IMS

SaaS




MQ


DataPower XG45

WXS, DataPower XC10

Public Cloud

Private Cloud

The Changing World: Budget Versus Architecture


Cast Iron, DP XH40

Enterprises looking to achieve “more with less”by better managing IT resources as collectives

IWD, PureApp

Integration Bus



WAS, CICS, IMS



Messaging backbone in the data center has extended to external clients connected via Internet

Messaging backbone in the data center has extended to external clients connected via the Internet

MQ, MessageSight


DataPower XG45

Internet

Mobile

Sensors

Scale and ubiquity of mobile and sensor-rich environments has changed requirements of the enterprise

WXS, DataPower XC10

Public Cloud

Private Cloud

IBM Mobile Foundation (Worklight)

The Changing World: And Then There Was Mobile


Cast Iron, DP XH40


IWD, PureApp

Integration Bus



WAS, CICS, IMS




MQ


DataPower XG45


Cast Iron, DP XH40

Public Cloud

Private Cloud


IWD, PureApp

Internet

Mobile

Sensors


Sterling CommerceDataPower XB62API Management

B2B partners opened channels and collaboration while a new genre “App Developer partner” is emerging

Partner community


Integration Bus, DataPower XI52

WXS, DataPower XC10


The Changing World: (Social) Evolution of “Partner”



WAS, CICS, IMS

Great…but what about SOA?

A repeatable business task –

e.g., check customer credit; open new

account

A Service

A way of thinking about your business through linked services and the

outcomes that they bring

Service Orientation

Service OrientedArchitecture (SOA)

An business-centric architectural approach based on service

oriented principles9

Partners Suppliers

DevelopersCustomers

APIs

Apps Patterns

Cloud Services

SOA mediates between consumers and providers (ESB pattern)

� 2005: Connecting and mediating in an IT transactional context

� 2010: Connecting and mediating e2e processes

� 2015: Connecting and mediating people, devices, Cloud, ….10


MQ

The World As We Knew It


WAS, CICS, IMS

12

Why is Messaging Important?

�Improvements in network reliability haven’t eliminated failures

�Failure handling still more effective and simpler to implement by separating it out from business and application logic

– Connectivity interruptions handled seamlessly without needing expert application integration skills

�Consistent interfaces, no matter the system, or the application programming language

– Removes complexity from application, reducing time and cost for development and maintenance

�Moves any type of data and any type of system, device or environment

�Can provide variety of qualities of service

– Persistence, non-persistence

– Point-to-point, publish/subscribe, multicast

– Move from batch and offline processing, to real-time event driven architectures

Synchronous Asynchronous

13

Build a universal messaging backbone

Use WebSphere MQ to flexibly integrate the pieces of your business

Connect remote devices, mobile enterprise and sensors

Harness the power of data outside your internal network

Application Connectivity

Moving data between applications, systems and services using MQI, JMS, REST, HTTP, Web Services, etc.

Enhanced end-to-end security

Securing your business data and helping meet industry regulations

Managed File Transfers

Improve reliability and security leveraging the internal messaging backbone

Ultra low latency messaging

Highest message throughput and lowest message latency over any transport

1

4

14

IBM WebSphere MQ 7.5 highlights

A fully integrated messaging

solution with secure, reliable and

scalable data movement for today’s

development needs

– Enhanced utilization of messaging through

use of Managed File Transfer solutions over the messaging backbone

– Access to end-to-end message encryption capability with a single install

– Increased entitlement to integrated

Telemetry capabilities reducing TCO of Mobile applications and smarter planet

technology

– Client applications gain access to

transactional integrity and updates across

all environments

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

WMQ

V7.5WMQ

Telemetry

WMQ AMS

WMQ MFT

Agent + Service

14

�WebSphere MQ becomes a single offering with a single install on distributed platforms, with the existing WMQ FTE and WMQ AMS code integrated as a part of the WMQ Server

Message Level Security

� “Valuable” messages

– In flight on the network

– At rest, on disk

– Monitoring and control messages

� Large networks, difficult to prove security of messages

– Injection

– Modification

– Unauthorized viewing

�Data subject to standards compliance (PCI, HIPAA, etc)

– Credit card data protected by PCI

– Confidential & personal data (government / healthcare etc)

– Data at rest, administrative privileges, etc

Message Level Security - Requirements

� Assurance that messages have not been altered in transit

– When issuing payment information messages, ensure the payment amount does not change before

reaching the receiver

� Assurance that messages originated from the expected source

– When processing control messages, validate the sender

� Assurance that messages can only be viewed by intended recipient(s)

– When sending confidential information

WebSphere MQ AMS – Signed Message Format (Integrity Policy)

Message Data

Message Data

PDMQ Header

PKCS #7 Envelope

Signature

Original MQ Message AMS Signed Message

Message PropertiesMessage Properties

WebSphere MQ AMS – Encrypted Message Format (Privacy Policy)

Message Data

Message Data

PDMQ Header

PKCS #7 Envelope

Signature

Original MQ MessageAMS Encrypted Message

Message PropertiesMessage Properties

Key encrypted with certificate

Data encrypted with key

How to secure an existing MQ application – No protection

STOCKSending

AppReceiving

App

ORDERSAlice Bob

How to secure an existing MQ application -SPLCAP(ENABLED)

STOCKSending

AppReceiving

App

ORDERSAlice Bob

1.Install WebSphere MQ 7.5 AMS Component on server

How to secure an existing MQ application – Assign Certificates

STOCKSending

AppReceiving

App

ORDERSAlice Bob

Keystore

Alice Private

Alice Public

Keystore

Bob Private

Bob Public


2.Create certificates (public / private key pairs)

How to secure an existing MQ application – Exchange Public Key

STOCKSending

AppReceiving

App

ORDERSAlice Bob

Keystore

Alice Private

Alice Public

Bob Public

Keystore

Bob Private

Bob Public

Alice Public



3.Exchange public keys

How to secure an existing MQ application – Set security policy

STOCKSending

AppReceiving

App

ORDERSAlice Bob

Keystore

Alice Private

Alice Public

Bob Public

Keystore

Bob Private

Bob Public

Alice Public

Policy

ORDERS

Privacy

Recipient : Bob

Signer : Alice




4.Define security policy for the queue

How to secure an existing MQ application – Privacy & Integrity

STOCKSending

AppReceiving

App

ORDERSAlice Bob

Keystore

Alice Private

Alice Public

Bob Public

Keystore

Bob Private

Bob Public

Alice Public

Policy

ORDERS

Privacy

Recipient : Bob

Signer : Alice




4.Define security policy for the queue

5.Messages can only be viewed by Bob, Bob will only accept messages from Alice

Send/RcvApp

Charlie

WebSphere MQ Advanced Message Security

� Provides additional security to that provided by base MQ

� Non-invasive

– No code changes or re-linking of applications

� End-to-end security, message level protection

– A security policy defines what protection should be applied to messages

� Asymmetric cryptography used to protect each message

– Integrity Policies prove message origin, content not changed

– Privacy policies as per integrity plus each message encrypted with unique key

� AMS Policies

–Integrity (End-to-end digital signing of messages)

–Privacy (End-to-end message content encryption)

How do most organizations move files today?

� FTP

– Typically File Transfer Protocol (FTP) is combined with writing and maintaining homegrown code to address its limitations

� Why is FTP use so widespread?

– FTP is widely available – Lowest common denominator

– Promises a quick fix – repent at leisure

– Simple concepts – low technical skills needed to get started

– FTP products seem “free”, simple, intuitive and ubiquitous

� Legacy File Transfer products

– A combination of products often used to provide silo solutions

– Often based on proprietary versions of FTP protocol

– Can’t transport other forms of data besides files

– Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure –especially with SOA

� People

– From IT Staff to Business staff and even Security Personnel

– Using a combination of email, fax, phone, mail, memory keys…

Most organizations rely on a mix of home-grown code,

several legacy products and different technologies …

and even people!

Shortcomings of Basic FTP

Limited Flexibility

Limited Security

Limited visibility and traceability

Limited Reliability

� Unreliable delivery – Lacking checkpoint restart – Files can

be lost� Transfers can terminate

without notification or any record – corrupt or partial

files can be accidentally used� File data can be unusable

after transfer – lack of Character Set conversion

� Often usernames and

passwords are sent with file –as plain text!

� Privacy, authentication and encryption often not be

available

� Non-repudiation often lacking

� Transfers cannot be monitored and managed centrally or remotely

� Logging capabilities may be limited and may only record transfers between directly connected systems

� Cannot track the entire journey of files – not just from one machine to the next but from the start of its

journey to its final destination

� Changes to file transfers often require updates to many ftp scripts that are typically scattered across machines

and require platform-specific skills to alter� All resources usually have to be available concurrently

� Often only one ftp transfer can run at a time� Typically transfers cannot be prioritized


A consolidated transport for both files and messages

• Traditional approaches to file transfer result in parallel infrastructures

‒ One for files – typically built on FTP‒ One for application messaging – based on

WebSphere MQ, or similar

• High degree of duplication in creating and maintaining the two infrastructures

• MQ MFT reuses the MQ network for managed file transfer and yields

‒ Operational savings and simplification‒ Reduced administration effort‒ Reduced skills requirements and

maintenance

File TransfersApplication

Messaging

Consolidated Transportfor Messages & Files


Handling both files and messages across MQ MFT

One file to one message

MQMFT

One file to a group of messages

One message to one file

A group of messages (or all messages on the queue) to one file

• File can be split based on:‒ Size

‒ Binary delimiter

‒ Regular expression

• One message becomes one file

• Optionally, a delimiter can be inserted between each

message used to compose the file

• One file becomes one message

MQMFT

MQMFT

MQMFT


Staged migration to messaging

� Pain-point:

– Hard to migrate to an event driven architecture as lots of applications

communicate by transferring files

� Managed File Transfer Helps:

– Deliver files as message payloads and vice versa

– Monitor queues and transfer message payloads to files

WebSphereMQ ManagedFile Transfer


Monitoring queues for the arrival of messages

� The WMQ MFT agent can

monitor queues for the arrival of

messages, then perform an

action, such as transferring the

payload from the messages as a

file (as per the previous slide)

� Conditions that can be monitored

for:

– Queue not empty

– Complete group of messages

ExistingApplication

WMQ

MFTAgent

ExistingApplication

WMQ

MFTAgent

Remember we said MFT

can monitor for files arriving…

Well, it can also monitor for

messages arriving on a queue…


Example usage of monitoring + program execution

1. Application writes

file to file

system

ExistingApplication

WMQ

MFTAgent

WMQ

MFTAgent

ExistingApplication

*tap*

2. Agent monitors file

system, spots arrival

of file and based on

rules, transfers the file

3. MFT transports file

to destination

4. At destination MQ MFT

writes file to file system

5. MFT can also start another

application to process the file


End-to-end encryption with WebSphere MQ Advanced(Advanced Message Security)

• MQ MFT already supports

transport level encryption using

SSL

• Data is encrypted before it is

sent over a channel and

decrypted when it is received

MQ MFT Agent

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

MQ MFT Agent

svrconnchannel

sndr/rcvrchannels

MQ MFT Agent

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

MQ MFT Agent

svrconnchannel

sndr/rcvrchannels

• The MFT and Advanced

Security of MQ Advanced

allows file data to be encrypted

at the source system and only

decrypted when it reaches the

destination system‒ This helps reduce encryption costs

‒ Data is secure even when at rest on a queue


• WebSphere Message Broker provides support for integration with IBM’s MFT suite to

transform, enrich file and route data

‒ E.g. Transforming and loading file data into a SAP system

• Broker adds intelligence to MFT networks

Message Flow

Integration Bus

FTEInput FTEOutput

WebSphere MQ

Message Flow

Agent

Agent

Agent

File andMessage data

External C:D network

C:DNode

C:DNodeC:D

NodeC:DNode

Internal C:D network

IBM Integration Bus and MFT


Protocol Bridging Agents� Support for transferring files located on FTP and SFTP servers

– The source or destination for a transfer can be an FTP or an SFTP server

� Enables incremental modernization of FTP-based home-grown solutions

– Provides auditability of transfers across FTP/SFTP to central audit log

– Ensures reliability of transfers across FTP/SFTP with checkpoint restart

� Fully integrated into graphical, command line and XML scripting interfaces

– Just looks like another MFT agent…

Audit information

Agent

WebSphere MQ

Agent Agent

ProtocolBridgeAgent

FTP/SFTP

FTP/SFTPServer

FTP/SFTPClient

FTP/SFTPClient

FTP/SFTPClient

Files exchanged between MFT and FTP/SFTP


MQ

The Changing World: Towards SOA

Integration Bus



WAS, CICS, IMS


Introducing IBM Integration Bus

� IBM’s Strategic Integration Technology– Single engineered product for .NET, Java and fully heterogeneous integration scenarios– DataPower continues to evolve as IBM’s integration gateway

Edge

IntegrationGateway

� A Natural Evolution for WebSphere Message Broker users– Significant innovation and evolution of WMB technology base– New features for Policy-based WLM, BPM integration, Business rules and .NET

� Designed to incorporate WebSphere Enterprise Service Bus use cases– Capabilities of WESB are folded in to IBM Integration Bus over time– Conversion tools for initial use cases built in to IIB from day one– WESB technology remains in market, supported. Migrate to Integration Bus when ready


� Broad range of operating system and hardware platforms supported– AIX, Windows, z/OS, HP-UX, Linux on xSeries, pSeries, zSeries, Solaris (x86-64 & SPARC), Ubuntu– Optimized 64-bit support on all platforms; 32-bit option available for Windows and x/Linux– New support for Windows 8 and Windows Server 2012; .NET CLR V4.5 included on Windows– Express, Standard and Advanced editions make IIB applicable for all solutions and budgets

• All new V9 features available in all editions unless otherwise stated

� Virtual images for efficient utilization & simple provisioning– Extensive support for virtualized environments, e.g. VMWare, AIX Hypervisor… any!– IBM Workload Deployer for x/Linux & AIX– Support for Pure on POWER hardware to complement xLinux– SmartCloud and IBM Workload Deployer images for simplified solution provisioning

� Includes access to full range of industry standard databases and ERP systems– DB2, Oracle, Sybase, SQL Server, Informix, solidDB– Open Driver Manager support enables new ODBC databases to be accessed– JDBC Type 4 for popular databases– SAP, Siebel, Peoplesoft, JDEdwards at no additional cost

� Technology components and pre-requisites– Java 7 on all platforms– MQ 7.5 prerequisite (7.1 on z/OS)

� Detailed System Requirements– Will be posted on www.ibm.com/integration-bus

A Broad Range of Supported Platforms and Environments

Traditional OS

IBM Pure

Private Cloud

IBM Workload Deployer

Public Cloud


432

Migration from WebSphere Message Broker V6.1, V7 and V8

� Migration from WMB V6.1, V7 and V8– All development assets (e.g. message flows, ESQL, DFDL, Java, Maps and XSLT) import directly

• Right-click convert action for pre-V8 maps; some manual tasks may be required– Migrate brokers using a single command, or create new brokers for phased migration

• No broker redeployment necessary when using built-in migrate command• All existing BAR files can be deployed to IB V9 brokers without change

� Migration commands for in-place migration– Includes migration of configuration data including broker databases, queues and registry– Forwards and backwards migration of existing components, in situ

• mqsimigratecomponents command (includes –t option for rollback to V7 and V8)

� Flexible co-existence options remove the need for additional hardware when migrating

– IB V9 co-exists on the same OS with all previous MB versions– MQ V7.5 required for all IB V9 brokers

• MQ V7.5 supported with all V6.1, V7 and V8 brokers for the purposes of V9 migration

1 Install IB V9 Stop broker Run migrate command Start broker

4321 Install IB V9 Create new broker Deploy existing assets Stop old broker

or


� Built-in conversion tools for WESB source assets– Initial emphasis on web services use cases (e.g. StockQuote)– Advanced use cases over time; convert when appropriate for your installation– Open framework for user and partner extensions

Conversion from WebSphere Enterprise Service Bus

� Simple workflow creates IB resources1. Export WESB PI from IID2. Import mediations into Eclipse Toolkit3. Right-click “convert” task to start conversion4. Follow guided editor to generate resources5. Task List will identify remaining manual steps6. Iterate as necessary

� Open Conversion Framework– Extensibility means more WESB primitives and resource types can be converted over time

• No minimum version requirement of WESB source• Builds directly into WESB conversion editor

– Design allows for future assisted resource creation from non-Integration Bus sources, e.g.– eGate Java collaborations and Event Type Definition, exploiting existing JAXB support– ICS collaborations, including ASBO and GBO model, exploiting new GDM pattern enablement


Web Visualisation and Analytics

� A comprehensive tool for web management– Manage all integration resources from zero-footprint client– Analyze integration performance in real-time– Supported on a variety of browsers: IE10, Firefox, Safari…– Complements MQ Explorer and WAS Admin consoles

� Managing Integration Resources– View top-level integration node properties– Add/remove/change integration servers– Start/Stop integration data flows – Role based access to control usage– Advanced options include data replay, policy & monitoring– Exploits underlying public REST/JSON API

� Integration Performance Analysis– Operational experience; no developer intervention required

• New and existing flows can exploit without change– Many metrics of integration flow available in real-time

• CPU & I/O time shown by default in integration analyzer• Other metrics include thread, data sizes, errors…

– Flexible display includes data tables and flow profile• Drill down to understand detailed behaviour

– Exploits underlying MQTT web sockets technology• Asynchronous notification at low CPU cost


Controlling Integrations with Policy

� Integration Workload Management– Provide intelligent mechanisms to control processing speed– Most common scenario is to reduce back-end server load– Design allows more policy-based processing over time– Can be applied to new or existing integration data flows

� Policy defines threshold limits and relevant actions– Set thresholds for integration data flow throughput– Specify actions at threshold, for example:

• NOTIFY: Higher (or lower) than threshold generates publication• DELAY: Excessive workload will have latency added to shape throughput• REDIRECT: Send excess to input node’s failure terminal or backout

� Web Console used to manage WLM policy– Sophisticated behaviour controllable by broker WLM policy– Workload can be managed across classes of message flows (e.g. batch vs. online)– Policies stored in local registry, and dynamically configurable– Developer can also specify limits as integration data flow properties

0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

160

180

200


� Target unresponsive flows through policy to improve overall system reliability– Additional WLM option aimed at unresponsive integration flows– An integration flow can become unresponsive for multiple reasons

• e.g. Waiting for external system, infinite loop, deadlock, malformed XML

� Flexible configuration, actions and reporting options– Specify threshold at which flows are considered unresponsive, e.g. 30 seconds for processing

• Configured via WLM policy, or directly on the flow in the BAR file– Define action to trigger when flow considered unresponsive

• Administrative notification through a new “timeout exceeded” event message– If flow eventually continues through to completion, a second event is published

• Restart the integration server (execution group) on which the unresponsive flow is running– New command option to forcibly stop integrations manually: mqsistopmsgflow –f

Managing Unresponsive Integration Flows

Waiting for

response


� Provide business insight during integration data flows– e.g. intelligent decision making; score then action in-flight request based on a business rule– User creates (e.g.) if-then-else rules using tool of choice (Excel, Word, Eclipse…)– The bus acts on these rules in flow, e.g. for business level routing

� New Decision Service node– Identifies inputs to business rules from in-flight data

• e.g. details of book order from request• e.g. the item price from key fields…

– Invokes built-in rule engine to perform business logic• Open interfaces for 3rd party and user engines

– Captures rules output for downstream processing• Business objects mapped back to in-flight data

� Create rules directly inside Integration Bus toolkit– Significant rules authoring facility built-in– Automatic package & deploy with integration assets– Dynamically reconfigure business rule using configurable service policy– Optionally refer to business rules on external ODM decision server– Exploit separate full ODM Decision Center for BRMS scenarios

� Embedded rules engine for high performance– Rule is executed in the same OS process as integration data flow

• Succeeds IAM9 Support Pac– Rule update notification ensures consistent rule execution– Optional governance of rules through remote ODM Decision Center

Synergy with ODM (Rules)


� IB provides powerful connectivity layer for BPM workflows

– Allows BPM developer to exploit rich integration features

• E.g. .NET, Healthcare Pack, TCP/IP, GDM, DFDL…

– No changes required to existing BPM programming model

• Helps maintain separation of concerns between roles

• Process designer works with integration developer

– Complements SCA nodes for BPM Advanced (WPS)

� Start with business process definition

– Process Center snapshots provides integration handover

• Snapshot can include multiple service definitions

• Captured as .twx file

– Integration developer imports snapshot from BPM

• Provides implementation of selected definitions

• Built-in integration tools simplify this activity (see below)

– Process designer re-imports updated snapshot from IB

• Completes business process definition

• Calls integration service in BPM system activity

� New BPM pattern simplifies creation of integration solution

– Start from Pattern Explorer, or right-click on existing service

• Import .twx file to create skeleton integration flow

– Customize created integration flow with IB capability…

• All other IB features available

– Deploy integration and pass back concrete references to BPM e.g. server IP address, etc.

Synergy with BPM (Express and Standard)


Comprehensive .NET Support

� New patterns and samples for MS Dynamics CRM and MSMQ– SAP CRM pattern for customer account synchronization

• Map account operations between BAPI & CRM Entities– Advanced CRM pattern enables dynamic graphical mapping– New customizable sample for 2-way MSMQ and MQ exchange

� New and enhanced nodes for .NET programmers– .NET Input node allows developers to initiate integration logic from any .NET system

• e.g. receive request from Dynamics CRM, AX, periodically read EXCEL file…• Highly customizable polling and trigger mechanisms

– CLR V4.5 runtime embedded within the integration server provides .NET technology foundation

• Languages include C#, VB .NET (COM), JScript & F#, with full range of .NET data types

• Also includes app domains for isolation

• Exploited by .NET Compute node and .NET Input node

– Further extensions include Visual Studio 2012, Windows 8/Server 2012 and Azure Cloud compatibility

� Developer Customizations– Cloned .NET nodes

• Easy to understand, consume and reuse

– Custom user properties• Expose key properties• e.g. CRM IP address

– Simple cloned node capture• User-defined icons• .NET Toolkit drawer


Integration Services for SOA

�Integration Services are well-defined containers of integration logic–Integration Services are created for and reside inside the bus–Interface is expressed via WSDL with a port type

• Interface and structure are both required

–e.g. Request, response & fault handlers per operation• Default binding is created out of the box

�Integration Services are defined through standard resources–WSDL (port type) defines service interface–Service interface defines one or more operations–Service Descriptor (XML) ties service interface with implementation–Each operation is implemented as an integration subflow

• Supporting resources also associated (e.g. Maps, XSDs)• Resources optionally reside in libraries

–Main entry point is implemented as an integration flow

�Simple lifecycle for services creation and management–Simple creation of new integration services

• Creating a new “Service” container• Import WSDL or create from scratch• Implement services• Specify binding before or during deployment

–Deployment as per standard integration applications–Unit Test and Team options available


� Service definitions allow you to make best use of available resources– Facilitates sharing of service information between users and systems– Allows users to understand interfaces (e.g. CustomerAddress.Update operation)– Provides a connector with which to exchange technical configuration (e.g. hostname)– Provides attachment points for associated policies (e.g. authorization)

� New framework enables discovery, cataloguing and re-use of services– Discovery connectors translate the service provider description to a common model– Interrogate IT systems for definition of technical assets – objects, functions and interaction points– User selects and refines definition of technical assets– Service definitions created and associated with technical assets– Discovered service definitions stored in embedded registry– Use catalogued services to configure integration solutions

� Initial implementation discovers and catalogs MQ service definitions1.Discover queues from referenced queue manager endpoint2.IB develops MQ service definition and stores in registry3.Use service definitions to configure MQ connectivity

–Sets required MQ headers and queue references

� New and existing nodes will be updated over time– Completely aligned with runtime connector framework

•Simple protocol points appropriate for style of interaction•Allows for simple development of custom connectors

MQ Service Discovery


� DBMS represents system of record for key business entities– Customers, accounts, partners, products… all stored in databases– Integration Bus tools discover and represent these key data– Integration services extends access to end-user applications

� New integration tools discover key database assets1.Connect to DBMS2.Discover source artefacts (tables, views, etc.)3.Map CRUD operations to service interface4.Save in canonical WSDL document

•Custom bindings for SQL access5.Re-use database WSDL in multiple scenarios

� Many uses for database service definition– WSDL contains both logical and physical database information– Drag and drop WSDL to automatically create SQL access methods– Create new integration service to exploit customized database access

•End-user application consumes as regular (e.g.) web service

� Customize integration services with data analysis– Tools for solving the problem of XML document understanding

•XML message formats can be structurally diverse•Often useful to semantically interpret related elements, e.g. healthcare CDA exchange format

– New Data Analysis Perspective provides a collection of useful data views•Model data based on input element XML; understand and visualise related elements•Generate resources (subflows, maps) that allow transformation between modelled elements

Database Service Discovery and Data Analysis

Products PartnersCustomers Accounts

© 2013 IBM Corporation51

Easy Data Modelling with DFDL

� Simple & powerful open standard for data modelling

– For use in IBM and non-IBM products

• e.g. Integration Bus, Rational Performance Test Server, Rational Test Virtualization Server, Rational Test Workbench, Rational Developer for System z…

• May also be used in standalone applications

– DFDL web community now active

• Public GitHub repositories for DFDL models

– Including HL7, ISO8583 and TLOG

• Commercial and scientific formats

• Collaborative development of models

� Support more features of the DFDL specification

– User-defined variables in DFDL expressions

– TLOG packed numeric fields

– Delimited binary data

– Fields lengths given by regular expressions

� Improved performance

– Continued improvement when parsing & writing

– Improved deployment times

� Usability Improvements to the DFDL editor

– Copy and paste of schema objects

– More refactoring operations


� IBM Graphical Data Mapper (GDM)

– Visually map and transform source to target data

– GDM designed for whole IBM product set, e.g.

• Integration Bus V9, WebSphere Message Broker v8, DataPower

• InfoSphere Master Data Management v10, Integration Designer v7.5/v8

• Rational Application Developer for WebSphere Software v8.5

• Rational Software Architect v8.5, RSA for WebSphere Software v8.5

• Other products yet to announce

– Rich feature set and simplicity make this a good default transformation choice

� Directly access stored procedures from within a map– Complements existing database select, insert, update, delete– Incorporate user-defined database functions into your graphical transforms– All standard broker databases supported, e.g. Oracle, DB2, SQLServer…

� Maps available to user patterns

– Graphical creation of flows which require transformation logic

• e.g. new input or output messages

– Invocation of mapper when pattern instances are generated

– User guidance through HTML pattern help and task list

– Patterns to demonstrate include CRM account mapping

� Migration of pre-V8 maps to IBM GDM– Most sophisticated maps can now be converted in a single step– Editor provides enhanced feedback about conversion to assist user understanding

Graphical TransformationsIBM Graphical Data MapperIBM Graphical Data Mapper

InfoSphere MDM RAD for WebSphere

IBM Integration Designer

IBM Integration Bus

Rational Software Architect

DataPower


Natural Integration with WAS Admin Console

�IIB is a compelling choice for WebSphere Application Server–WAS provides efficient application development and delivery–New tools to simplify learning curve for WAS users–Addresses administrator requirements–Supported on WAS V7 and V8.x

�WAS Admin Console Broker Plug-in–Varied set of MB administration tasks available

• Connect to multiple local or remote brokers• View available execution groups and their current status• View services, applications, libraries, message flows • View Message Broker console help topics

–Uses standard features for ease of configuration• Role-based access to prevent unauthorized administration• REST APIs for local and remote management

–Use WAS Admin Console for WAS centric administrator• Complements IIB Web UI for IIB-centric administrator

–Design allows for future modification of integration resources, start, stop etc.

�Fully compatible with WAS ND–Integration feature operates at cell level–The configuration data is stored by the Deployment manager–All Application Servers have access to cell level configuration data


� Provides basic mapping, routing and versioning capabilities to applications– Scenarios include service versioning and meet-in-the-middle interfaces – Uses existing developer skills and tools, e.g. Java and RAD– Service Mapping pipeline invoked directly from application– Delivered as part of WebSphere Application Server– Service Mapper can generate events for Integration Bus

• Enables audit, replay and out-of-band processing

� Developer experience enables mapping between Application Server services– Line-of-business application developer calls service from within application logic– New WAS mapping service intercepts, maps and reroutes WS calls accordingly– Uses existing application developer tools

• Develop service maps directly within RAD using standard GDM technology• Configure and enable in WAS Admin Console

– Map between services without leaving WAS runtime environments for high efficiency• Included in WAS v8.5.5 – no additional install or license required

� IIB consumes events from service mapping for “system of awareness”– Service map invocations can emit business events

• Just like IB integration flows, can be captured, recorded and replayed– Built on GDM technology with interface mapping

• Allows for future hosting of same service maps inside Integration Bus

WAS Service Mapping for Application Developers

APP APP

Service MappingPipeline

App Server

Events

Integration Bus


� Introducing IBM MessageSight– An appliance-based messaging server built on special purpose hardware– Supports very large numbers of connected clients and devices, and high volumes of messages– Secures the edge of the enterprise and enables use cases like mobile and telemetry

� Two new patterns for integrating IBM MessageSight with backend systems– Covers common use cases for bi-directional connectivity– Use of JMS enables standards-based appliance connectivity that is also extensible to other providers– Pattern design allows for future selection of high performance, standard MQTT as protocol

� 1) Event Filter Pattern– Messaging appliance routes inbound events into the broker via JMS– The broker narrows down events using decision service and inserts the subset into backend systems

� 2) Event Notification Pattern– The broker detects an event from a backend system (e.g. message queue, database trigger)– Broker fans out event via JMS to the appliance to interested connected clients

Easily Integrate with Appliance-based Messaging

Me

ss

ag

eS

igh

t

Many connected

devices

IB Back-end system

Filter

Me

ss

ag

eS

igh

t

Many connected

devices

IB Back-end system

Notification


Mobile App Platform

� Four new IIB development patterns

− Fully integrated in IIB toolkit

− Generate Worklight adapter, test application and supporting WMB flows

1. Simple Service to mobile – WSDL based

2. Resource access from mobile - Controlled access to enterprise data as a resource: methods for read create, update and delete

3. Microsoft .Net service pattern

4. Queue based Push Notification

� Patterns exploits value add IIB capability including policy decision point for access authorization and WebSphere eXtreme Scale for performance

� Mobile enable any enterprise service in 2 clicks !

� Build robust solutions with integrated caching and security

� Push data to mobile users from enterprise applications

� Create end-to-end mobile solutions for Microsoft .NET

Integration Bus and Mobile Integration


Global Cache Enhancements

�IB contains a built-in facility to share data between multiple brokers–Improve mediation response times and dramatically reduce application load–Typical scenarios include multi-broker request-reply and multi-broker aggregation–Uses WebSphere Extreme Scale coherent cache technology

�Support for external software and hardware caches–Access separate eXtreme Scale and DataPower XC10 appliances from within the broker–Allows broker to interact with enterprise caching solution without embedding additional libraries–Cache access, activity log, resource statistics etc. just like embedded cache–Operationally configured using dynamic configurable service–New EG options to specify SSL connections to external WXS grids

•Uses existing MB SSL infrastructure to configure certificates

�Cache Expiry options–New getGlobalMap() variant to set the time to live for data in the embedded global cache.

•MbGlobalMap evictMap = MbGlobalMap.getGlobalMap("…", new MbGlobalMapSessionPolicy(30));

•evictMap.put("key", "val");

–Specify a value in seconds. The default value is 0, which means data never gets automatically removed.

�Programming and operational enhancements–Insert and lookup map data using a wider range of Java object types for simplified programming logic–Support for highly available multi-instance configurations

MyVar = Cache.Value;

Broker1 Broker2

Cache.Value = 42;

External Cache

Accelerating Integration with Built-in Patterns

• Can be quickly configured and deployed, and extended to create new patterns

• Patterns for SAP, mobile, BPM, Message-based integration and more!

Service Enablement

Service Virtualization

Application Integration

Encapsulating functionality and

presenting it through a service-

oriented interface

Loose coupling and mediation

between services

Enabling rapid integration of

applications across the enterprise

e.g. SAP

Message-based Integration

File Processing

Custom Patterns

Providing routing, transformation

and logging services for

messages

Managed runtime environment for

processing files such as shredding

into individual transaction records

Create your own patterns to re-

use across your organization

Best practice, reusable solutions to common integration problems

Questions and Answers …




MQ, MessageSight


DataPower XG45

Internet

Mobile

Sensors


WXS, DataPower XC10

Public Cloud

Private Cloud


The Changing World: And Then There Was Mobile


Cast Iron, DP XH40


IWD, PureApp

Integration Bus



WAS, CICS, IMS

Mobile … a new frontier in business growth

� Mobile B2C

• Increase customer satisfaction by enabling banking, insurance, and trading anywhere, anytime

• Reach customers in new ways through mobile applications, SMS, email

� Mobile B2E & B2B

• Enable field employees for increased productivity

• Greater efficiency and accuracy in supply chain operations

• Exchange business information with partners securely

� (Mobile) M2M

• Enable the exchange of data and events between businesses and machines

• Internet of Things - sensor events feeding information and driving a smarter planet

Why messaging for mobile and machine-to-machine?

� HTTP revolutionized how we consume data

– Simple request/response model

– Available on any tablet, laptop, phone, PC

– Not designed for wireless

– Slow and unreliable on mobile networks

� Mobile and M2M applications have additional challenges

– Requires a real-time, event-driven model

– Publishing information one-to-many

– Listening for events as they happen

– Sending small packets of data in huge volumes

– Reliably pushing data over unreliable networks


Use case – Connected car

Connected car

vibration detected, details

published

Unlock my car

schedules appointment

with car owner

Find my car

predicts part failure


MQTT – the protocol for mobile and M2M messaging

� MQTT is a messaging protocol ideally suited to mobile and M2M environments

� It has an open specification (http://mqtt.org)

– There are over 40 different client implementations

– Standardization with OASIS is in progress

� High-quality, open-source implementations of clients

– Hosted at the Eclipse Paho project

– Build the clients yourself or use free ones from IBM or others

� MQTT is very lean and fast

– Efficient format with minimal overhead

– Client implementations are small and can run on small devices

� Communication using messaging is much more flexible than request/response

– Bi-directional, asynchronous “push” communication

– Publish/subscribe decouples the senders of information from the receivers


Introducing IBM MessageSight

� IBM MessageSight is a messaging appliance

� Allows your business to scale to the demands of the mobile and M2M use cases

� Easily and securely extends your existing messaging infrastructure across the

Internet

� Enables real-time, streaming analytics and event processing


IBM MessageSight feature summary

• Appliance form-factor simplifies physical installation

• Up and running in under 30 minutes

• Task-oriented UI guides the administrator through the first steps

• Simple and scalable administration using policies

• Fast time-to-value



DMZ ReadyAppliance

Secureand Reliable

• Hardened appliance form-factor suitable for DMZ use• Secure firmware• No user-visible, general-purpose OS

• SSL/TLS support for MQTT, JMS and MQ connections

• Reliability• Supports both nonpersistent and persistent messaging• Hardware-assisted persistent store

• High availability using pairs of appliances• Synchronous data replication from primary persistent store

to standby



M2M and Mobile

Optimizedfor Wireless

ClientsDMZ ReadyAppliance

Secureand Reliable

• Efficient MQTT protocol ideal for mobile and M2M deployments

• Large range of clients supported including JavaScript using WebSockets

• Apps can be HTML5 web apps, native or hybrid

• Integrates easily with IBM Worklight



M2M and Mobile



Secureand Reliable

Simple Programming

Model

DeveloperFriendly

• Simple yet powerful APIs consistent across a wide choice of languages

• Simple programming model (connect, publish, subscribe)

• Active development community on IBM developerWorkshttp://www.ibm.com/developerworks/connect/IBMmessaging



M2M and Mobile



Secureand Reliable

Simple Programming

Model

DeveloperFriendly

Open Standards

Easy to Integrate

• Compatible with a wide variety of environments• JMS for Java Standard Edition environments• WebSockets for Rich Internet Applications• MQTT protocol with many open-source clients

• Built-in connectivity with WebSphere MQ• One appliance can connect to multiple queue managers• Destination mappings to/from MQ queues/topics

• Integrates with IBM Integration Bus (Message Broker)• New mobile patterns integrate using JMS nodes



Orders of Magnitude

Improvements

MassiveScale

M2M and Mobile



Secureand Reliable

Simple Programming

Model

DeveloperFriendly

Open Standards

Easy to Integrate

• A million concurrently connected devices

• 13 million nonpersistent messages per second

• Over 400K persistent messages per second

• Predictable low latency

• Carefully optimised design


MessageSight solutions for Automotive

� What are they interested in doing today?

� Offering new innovative services to the drivers, dealers, partners, etc

� Find my car / unlock my car

� Predictive analysis for parts failure

� Impose limits on policy and driver(who can go where at what speed..)

� Unattended car rental

� eCall 2015 (EU obligatory emergency call capability in vehicle)

� Automated driving system

�

� What problems are they facing?� Connect to millions of vehicles gathering vast amounts of telematic data in real time, analyze the data, and determine

message response back to a single or multiple vehicles or other interested parties

� Ability to store messages for vehicles that temporarily lose communication

� Integrate with existing data sources in enterprise systems such as vehicle service history records

� Integrate with partners

� What is needed?

� Efficient and reliable two way communication

� Access security to ensure information is only sent between authorised vehicles and trusted sources

� Open protocols (WebSockets, JMS, MQTT) and MQ Connectivity

for easy integration to both the internet and the enterprise


Internet

APP

APP

Service

ServiceDB APP DB

APP

DB

WebSphere MQ or Appliance•Universal Enterprise Messaging

•Provide access to enterprise apps

and data already connected by MQ

•Pub/sub model to enable dynamic

distribution of notifications

Apple iOS

Messaging for

native iOS apps

JavaScript

Messaging for

Hybrid apps

Android

Messaging

for native

Android apps

MQTT•Reliable messaging

protocol

•Access to enterprise

•Reliable delivery

•Conserve device

power

•Reduce network traffic

WorklightdevelopersDeveloper experience augmented with mobile messaging

JavaScript*

Messaging for

Web apps

Available from new IBM Messaging Community on developerWorks

Messaging Clients for Mobile Apps

* = beta only

Clients for Mobile and M2M MessagingIncluding Simple Javascript Messaging API




MQ


DataPower XG45


Cast Iron, DP XH40

Public Cloud

Private Cloud


IWD, PureApp

Internet

Mobile

Sensors


Sterling CommerceDataPower XB62API Management

B2B partners opened channels and collaboration while a new genre “App Developer partner” is emerging

Partner community


Integration Bus, DataPower XI52

WXS, DataPower XC10


The Changing World: (Social) Evolution of “Partner”



WAS, CICS, IMS

What is a Web API?� A web API is a public persona for an enterprise; exposing defined assets,

data or services for public consumption

� A web API is simple for app developers to use, access and understand

� A web API can be easily invoked via a browser, mobile device, etc.

What Value Does a Web API Provide?� Extends an enterprise and opens new markets by allowing external app

developers to easily leverage, publicize and/or aggregate a company’s assets for broad-based consumption

What “assets, data or services”are exposed via an Web API?:� Product catalogs

� Phone listings

� Insurance cases

� Order status

� Bank loan rates

The Basics of a web API

ExternalApp Developer

Apps, APIs and API Mgmt…

Business Owner IT

Developer

Consumers

New business opportunities

• New markets

• Increase customers

• Enhance branding

• Competitive advantage

Extend development team

•Increase innovation

•Increase scale

Partner/supplier

alignment

BenefitsBenefits

ChallengesChallenges

Business strategy

Infrastructure

• Security

• Creation

• Scalability

Operational control

• Publish

• Analyze

• Monitor

Easily Define APIs

• Define the API you wish to expose

• Then configure the API by proxying

an existing REST API or assemble a new API

• Provide examples of the request and response messages, headers

and parameters

Assemble New APIs Through Configuration

• Connect to one or more datasource‒ DB2

‒ MySQL

‒ SQL Server

‒ Oracle

‒ Salesforce.com

‒ SOAP to REST

‒ HTTP

• Drag and connect linking the request and response messages

• Transform the message elements with a click



MQ


DataPower XG45

WXS, DataPower XC10

The Changing World: Emergence of Appliances

Integration Bus



WAS, CICS, IMS

Secure integrationSecurely integrate API, Web & Mobile workloads, in addition to SOA & B2B

Mobile-ready security gatewaySecure & optimize delivery of Mobile applications & integrate with IBM Worklight

Faster consistent response timeReduce load on back-end systems and optimize delivery through local & external caching and intelligent load distribution

Secure. Integrate. Optimize.

Pattern-based configurationCreate & deploy common configuration patterns for reduced time to value, improved productivity & quality

Deployment flexibilityUse physical or virtual appliance with seamless configuration migration

System z integrationEasily consume external web services from IMS & expose IMS data as a service

DataPower 6.0

DataPower in a Nutshell

Enable additional use-cases with a single, policy-driven converged gateway

Web Service

Gateway

• DMZ-ready

• Security gateway (AAA, XML threat)

• Service level management and monitoring

• Intelligent load distribution & dynamic routing

Integration Gateway

• “Any-to-Any”conversion at wire-speed

• Bridges multiple transport protocols

• Mainframe integration & enablement

B2B Gateway

• DMZ-ready

• B2B Messaging (AS1/AS2/AS3/ebMS)

• Trading Partner Profile Management

• B2B Transaction Viewer

Web Application

Gateway

• DMZ-ready

• First-class integration with WAS

• Cache response content

• Web application security

• Traffic mgmt

On-premiseAPI

Management

• DMZ-ready

• Web API security

• Monitor API use

• Enforce API consumption policies

Mobile

Application

Gateway

• DMZ ready

• Mobile application security

• Support Worklightmobile platform

• Monitor and control mobile app access

Focus so farFocus so far Expanded focusExpanded focus

Form factorsForm factors

Physical Appliance for hardware performance & security

Virtual Appliance for deployment flexibility

DataPower Classic Use Cases

• Monitoring and control

– Example: centralized ingress management for all Web Services using ITCAM SOA

• Deep-content routing and data aggregation

– Example: XPath (content) routing on Web Service parameters

• Functional acceleration

– Example: XSLT, WS Security

• Application-layer security and threat protection

– Example: XML Denial-of-Service protection, WS Security, AAA

• Protocol and message bridging

– Example: Convert to WS to legacy Cobol/MQ

Service Providers

Clients

In-the-clear SOAP/HTTP

MaliciousSOAP/HTTP

SOAP

SOAP

SOAP

Cobol/MQ Appl

Cobol/MQ

Encrypted and Signed SOAP/HTTP

In-the-clear SOAP/HTTP

Use Case: Security Gateway (Consumer or Employee)

AAA authenticates user and ensures they are authorized to the resource beings accessed

Usually (REST) JSON or XML over HTTP from web or mobile application

XML and JSON well-formedness, schema valication and DoSprotection

SLM shapes and throttles traffic based on seggregatedconsumer types

Request are transformed to internal schema and routed appropriately

Responses are cached to improve response time

XG45

DMZ

Trusted

ZoneInternet

Use Case: Security Gateway (B2B)


Usually Web Service (SOAP) or AS* (XB62) protocol

XML well-formedness, schema valication and DoSprotection




DMZ

Internet

XG45

Trusted

Zone

Use Case: Security Gateway (Inter-LoB)


Usually Web Service (SOAP) or AS* (XB62) protocol

XML well-formedness, schema valication and DoSprotection




SecurityZone

BoundaryTrusted

Zone A

XG45

Trusted

Zone B

Use Case: Integration Gateway (Edge of Service)


Usually Web Service (SOAP) or REST

Requests are transfomed to local format (eg. Cobol)

SLM throttles all inbound traffic to perform overload protection of backend resource

Request are transmitted over local transport and intelligently distributed (e.g. MQ)


XI52

Trusted Zone

IBM Messaging Focus Areas

Deliver Messaging Backbone for EnterpriseFocus on traditional MQ values, rock-solid enterprise-class service, ease-of-operation, breadth of platform coverage, availability, z/OS exploitation

Capture Big Data from Mobile and Internet of ThingsFocus on Internet-scale events, m2m device enablement, zero-admin, security and privacy, feed into real-time analytics, location-based notifications

Enable Developers to build more scalable, responsive applicationsFocus on new app dev use cases, breadth of languages, ease-of-deployment, lightweight services, integration with developer frameworks

Questions and Answers …

ibm connectivity and integration

Technology

ibm corporation

performance of applications

integration gateway

ibm connectivity

ims application platforms

changing world

heritage services wxs

datapower xc10 cache