fundamentals of websphere mq securitymqtechconference.com/sessions_v2013/...websphere_mq... ·...
TRANSCRIPT
Capitalware's MQ Technical Conference v2.0.1.3
The top ten issues in
WebSphere MQ and
WebSphere MB
Barry D. Lamkin
Executive IT Specialist
Capitalware's MQ Technical Conference v2.0.1.3
Who Am I?
Barry Lamkin
Army Helicopter Pilot 1967 – 1971
Air Traffic Controller 1973 - 1981
MVS (aka z/OS) Systems Programmer 1981 – 1994
Candle Systems Engineer 1994 – 2004
IBM Executive IT Specialist 2004 – whenever
Capitalware's MQ Technical Conference v2.0.1.3 3
WebSphere MQ (MQSeries) - Features
Assured, exactly once delivery
Single API across 45+ platforms
Network integration across various
network protocols
Transactional control
Triggering of jobs/programs
Content independence
Single message > 100MB
Asynchronous design (application
& platform independent)
Parallel processing
Robust, commercial middleware
Shields developers from
network complexities
Capitalware's MQ Technical Conference v2.0.1.3
Monitoring Points used to
track transaction flow
A
B
C
Queue Manager QM1
Queue Manager QM2
Application A, B and C are components of a business transaction
There are many objects that make up the underlying transactions
There are as many as 8 monitoring points for objects associated to this transaction
You should Monitor all the objects as it relates to the transaction (queues, channels)
You should Monitor the flow of messages belonging to specific message queues (get/put rate)
You need the ability to detect slowdown or stoppage in flow in specific queues (put rate exceeds get rate,
channel down)
You will need to identify problems and automatically react to them.
QB QC
QA
TQ
TQ
Capitalware's MQ Technical Conference v2.0.1.3
Top Ten Issues in WMQ & WMB (maybe 11)
MQ Channel Down
Queue Full
Messages in the Dead Letter Queue
Messages in a queue and no open processes
Isolating MQ problems between IBM z/OS® and
distributed systems
Changes in the MQ configuration
Restoring last valid MQ configuration
Determining if slow performance is due to network, MQ or
Message Broker
Problems connecting to broker's queue manager
No messages flowing in the broker
Execution Groups or Message Flows not started
Capitalware's MQ Technical Conference v2.0.1.3
MQ Channels
Capitalware's MQ Technical Conference v2.0.1.3
Issues with Channels
Which channels are running or stopped?
If my channel is up, is it transmitting
messages?
Are my channels optimally configured?
If channel performance is poor, how does that
impact my clients?
Capitalware's MQ Technical Conference v2.0.1.3
MQ Channel Down
Is this a problem?
MQ Event “Channel Stopped” may not be good enough
Check Channel Status
Are there messages in the XmitQ?
Is the Remote Queue Manager down?
Is the Remote Queue correct or full?
Is there a Network problem?
Capitalware's MQ Technical Conference v2.0.1.3 9
Channel Stopped
The channel is stopped. We note that we should have had a situation that triggers an alert when a current channel is not in running status, so that this whole scenario could have been avoided automatically for an Xmit queue.
Capitalware's MQ Technical Conference v2.0.1.3 10
Start the Channel
• Right click on the channel row and select the Take Action option, then select the action to start the channel. The channel name for the selected row is automatically filled in.
• If we had the situation to detect the stopped channel, the action to start it could be automated to occur at the time the issue is detected.
• Note that any MQSC command can be configured as a take action, even if it is not product-provided.
Capitalware's MQ Technical Conference v2.0.1.3
Queues
A queue is a container for messages Local Queues
Transmission Queues
Remote Queues
Alias Queues
Model Queues
Managed by the Queue Manager
Queue Defined Predefined
Dynamically defined
Messages are placed in queues to allow programs to interact
with each other asynchromously
Capitalware's MQ Technical Conference v2.0.1.3
Queue Statistics / Definitions
Active, Input and Output Processes
Local Aliased Queue Name and Type
Queue Configuration
Trigger Active
Trigger Depth
Max Message Length
Capitalware's MQ Technical Conference v2.0.1.3
Queue Full
Local Application or MCA can no longer put messages to Queue
Local Application should check for Queue Full condition MQRC_Q_FULL (2053, X'805')
MCA will put messages to Queue for a Remote Application If Queue Full messages go to Dead Letter Queue
If no Dead Letter Queue channel will be stopped
MQ Event “Queue Full” or “Queue High” may not be adequate These events do not get reset until the Queue Depth hits “Queue Low”
Check Queue Depth
Check if messages are on a Queue and no processes have it opened
Capitalware's MQ Technical Conference v2.0.1.3 14
Queue Depth High
Situation automatically triggers alert about queues with high depth.
Here, two queues are full, but usually you would set the situation to trigger for high depth in time to avoid the critical full queue.
Capitalware's MQ Technical Conference v2.0.1.3 15
Check Queue Status
Note how two applications have it open for output, but none for input.
There are some really old messages on this queue.
Since this is an Xmit queue, the channel should have it open for input.
So alternatively, a situation could have automatically detected that the queue is not open for input though it has a depth > 0.
Capitalware's MQ Technical Conference v2.0.1.3
Issues with DLQ
Are there any messages in the DLQ?
How long has a message been in the DLQ?
What messages are in the DLQ?
Why is a message in the DLQ?
Can I view the message?
Can I delete one or more messages?
Can I requeue a message?
Capitalware's MQ Technical Conference v2.0.1.3
Messages in the Dead Letter Queue
Dead Letter Queue prevents the Queue Manager from
stopping the channel
Need to monitor if messages arrive in the Dead Letter
Queue
Need to quickly isolate the cause of the message(s)
arriving in the DLQ
Need to be able to resolve the issue with the DLQ
messages Fix the issue
Delete the message(s)
Retry the messages
Capitalware's MQ Technical Conference v2.0.1.3
MQ problems between IBM z/OS® and
distributed systems
MQ provides common API across all platforms
Different monitoring/management solutions for each
platform?
Best practices dictate looking at MQ environment
holistically
Differing backup & recover strategies for MQ objects
Capitalware's MQ Technical Conference v2.0.1.3
MQ Configuration - Administration
Enterprise-wide configuration/administration strategy?
Different strategies depending on platform?
What about backup/recovery?
If an object is modified you need to be able to detect
that.
Capitalware's MQ Technical Conference v2.0.1.3 20
WebSphere MQ Configuration Management
Navigator
showing WMQ
Configuration Same interface for monitoring and
configuration of WMQ across both
distributed and zOS environments
Configure and deploy Queue
managers from here
Discovery of existing MQ resources
Prototyping/Modeling (templates) of
configuration
Validation prior to deployment
Deployment of template based configuration
to multiple nodes, immediate or scheduled
Synchronization from actual to defined or
defined to actual
Audit trail
Identify WMQ configuration discrepancies
Recovery of WebSphere MQ Environments
Capitalware's MQ Technical Conference v2.0.1.3
WebSphere Message Broker (IIB)
More of a Black Box than MQ Execution Groups
Message Flows
Processing Nodes
Threads
Need to understand if MB is being affected by MQ
Is the Queue Manager running?
Is the Broker connected to the Queue Manager?
Are the issues with the Network?
Are there issues with the OS?
Demands a holistic approach to monitoring to improve MTTR
Capitalware's MQ Technical Conference v2.0.1.3
22
Message Broker Performance & Availability Monitoring
WMB Reports
Navigator with
WMB monitor Same interface, monitoring paradigms for
managing both WMQ and WMB across
both distributed and zOS environments
Monitor message broker / message flow events
Provide performance statistics at broker, execution group,
message flow, sub-flow and processing node levels
Report pub/sub neighbors, subscriptions, retained
publications and ACL entries in broker
Detect broker configuration changes
Message broker operational command access
Detect Broker Status - stopped broker, problems
connecting to broker’s queue mgr, stopped message
flow, an active trace, expired publications or
subscriptions
Track Broker performance – message flow time, queue
time, input/output rates
Capitalware's MQ Technical Conference v2.0.1.3 23
Message Flow Workspaces
Message Flow Accounting Workspaces.
-Message Flow Throughput: shows the Total
Input Message,. Total Committed, Total
blackout.
-Message Flow Start Rate: shows the Total
Input Message Rate
Recent/Historical Message Flow Accounting Workspaces
-Recent/Historical Message Flow Throughput: shows the
Total Input Message, Total Committed, Total blackout
-Recent/Historical Message Flow Start Rate: shows the Total
Input Message Rate
Capitalware's MQ Technical Conference v2.0.1.3 24
The Message Broker
Explorer enables you to
start/stop message flow
statistics on the broker,
and view the output.
New in V7 (although
supportpac IS02
available for V6.1)
Warnings are displayed
advising there may be a
performance impact
(typically ~3%)
Broker Explorer & Message Flow Statistics
Capitalware's MQ Technical Conference v2.0.1.3
What else? Logs (different for z/OS and Distributed)
Buffer Pools (z/OS)
Page Sets (z/OS)
Shared Queues/Coupling Facility (z/OS)
Clusters
Age of Messages
Queue/Dequeue Rate
Channel Usage
Poison Messages
What’s normal activity?
Predictive Analysis
Capitalware's MQ Technical Conference v2.0.1.3
Buffer Pool Statistics - zOS
Capitalware's MQ Technical Conference v2.0.1.3
Channel Initiator Status - zOS
Capitalware's MQ Technical Conference v2.0.1.3
Log Manager Performance - zOS
Capitalware's MQ Technical Conference v2.0.1.3
Message Manager Performance - zOS
Capitalware's MQ Technical Conference v2.0.1.3
Page Set Statistics - zOS
Capitalware's MQ Technical Conference v2.0.1.3
Summary
Once introduced into an environment WMQ and WMB become
ubiquitous.
Monitoring/managing your messaging backbone in a silo is not
adequate
Start looking at your Enterprise Monitoring & Management strategy
holistically.
It may look like the problem is WMQ or WMB, maybe it Is not.
Use automation for corrective action when appropriate
Capitalware's MQ Technical Conference v2.0.1.3
Questions & Answers