Skip to content

IBM MQ compendium

IBM MQ is the enterprise solution to exchange message over queues.

As it supports loosely coupling communication between applications, via asynchronous protocol, and message exchange, it has to be part of any modern digital, responsive solutions, and so it makes sense to write about it in the context of EDA.

It is used for years to support asynchronous communication in mainframe applications.

Overview

A queue manager can be thought of in two parts: the data stored on disk, and the running processes that allow access to the data.

Queue managers can be connected together via network channels to allow messages to flow between disparate systems and applications on different platforms including on-premise and cloud systems

MQ value propositions

  • No data loss, no duplicate
  • Integrate with transaction
  • Scale horizontally: add more queue managers to share tasks and distribute the messages across them. MQ Clusters will even intelligently route messages to where they’re needed. The world is full of horizontally scaled MQ systems that handle billions of messages a day.
  • High availability with replicated Queue managers. Active/active horizontal scaling for always on systems
  • Lightweight and scale to run in any size
  • Containerized, runs in orchestration like OpenShift

Comparing MQ Pub/Sub and Kafka

Assess the following characteristics:

  • Event History: does the solution need to be able to retrieve historical events either during normal and/or failure situations? If so for how long and how much data?
  • Fined Grained Subscriptions: Should applications receive all event types from a topic? (Even events that are irrelevant to them ) With MQ topics, they are hierarchical, so it is possible for consumer apps to subscribe to different level of the topic. With Kafka topics are partitioned and consumers get all the events from one to many partition.
  • Scalable Consumption: if 100 consumers subscribe to all events on a topic, IBM MQ will create 100 messages for each published event (with the exception of multicast Pub/Sub). Each of event will be stored and, if required, persisted to disk using system resources.
  • Transactional Behavior: With pub/sub thee tx requirement is less critical, than in queues

High Availability

High availability for IBM MQ in containers proposes 3 HA configurations.

See my summary here

  • Native HA queue managers involve an active and two replica Kubernetes Pods, which run as part of a Kubernetes StatefulSet with exactly three replicas each with their own set of Kubernetes Persistent Volumes. It only runs in the same k8s cluster.

Any queue manager can be moved to a different Kubernetes Node, as long as it keeps the same data (provided by Kubernetes Persistent Volumes) and is still addressable across the network by client applications.

If the PVs are served via a SAN then there will be replications done at the SAN level too.

  • Multi instance active - standby topology with PV RWM settings.

Clustering and distributed queuing

Distributed queuing means sending messages from one queue manager to another. Clustering QM simplifies the communication configuration between QMs.

Each queue manager has a definition for each of its queues (local or remote).

If the messages are destined for a remote queue, the local queue manager holds them on a transmission queue, which persists them in a message store, until they can be forwarded to the remote queue manager.

Each queue manager contains communications software, known as the moving service, that the queue manager uses to communicate with other queue managers.

A channel is a one-way communication link between two queue managers.

The software that handles the sending and receiving of messages is called the Message Channel Agent (MCA). There is a message channel agent (MCA) at each end of a channel.

Replicated Data Queue Managers

MQ Advanced supports synchronous replication and fast quorum based take over for HA scenarios, recovery in seconds. It also supports asynchronous replication between quorum groups to support long distance DR deployments.

All nodes support concurrently running multiple different active queue managers with bidirectional asynchronous replication, supporting active/active HA and DR topologies.

Use queue manager leader and two replicas. So messages are replicated in three locations. Those are exact replicas, maintaining configuration, message order, transactional state. Quorum ensures consistency and rapid failure (within a second) and recovery.

The MQ operator helps to deploy MQ manager with declarative manifest, with native HA, cross availability zones, with all the networking services, and storage needed. Applications connected to MQ manager do not know how many queue managers are behind their request.

Always on MQ

MQ provides a uniform cluster consisting of multiple active queue managers acting as a single messaging service.

Each Queue manager has the same resource, queues, channels...

Applications can connect to any of the queue managers within the uniform cluster. This removes any dependency on a specific queue manager, resulting in better availability and workload balancing of messaging traffic.

The queue managers are configured almost identically, so that an application can interact with them as a single group.

Insight to your data

Stream MQ data by adding a new queue and specify the original queue the name of the streaming queue

MQSC commands

Use MQSC commands to manage queue manager objects, including the queue manager itself, queues, process definitions, channels, client connection channels, listeners, services, namelists, clusters, and authentication information objects.

  • Example of common commands: (start a bash in the docker image and use runmqsc tool)
display queue(rawtx)

Those MQSC commands can be define in a config map to be loaded inside the docker image. See example of such map here

apiVersion: v1
kind: ConfigMap
metadata:
  name: mq-mqsc-config
data:
  example.mqsc: |
    DEFINE QLOCAL('ITEMS') REPLACE
    DEFINE CHANNEL('DEV.ADMIN.SVRCONN') CHLTYPE(SVRCONN) REPLACE
    DEFINE QLOCAL('DEV.DEAD.LETTER.QUEUE') REPLACE
    ALTER QMGR DEADQ('DEV.DEAD.LETTER.QUEUE')
    DEFINE CHANNEL(DEV.APP.SVRCONN) CHLTYPE(SVRCONN) 
    ALTER QMGR CHLAUTH (DISABLED)
    REFRESH SECURITY TYPE(CONNAUTH)

AMQP

Advanced Message Queuing Protocol is a standard to integrate with messaging product. IBM MQ supports AMQP 1.0.

Build custom MQ docker image

Here are the step to build a custom image for MQ

  • Clone https://github.com/ibm-messaging/mq-container.git
  • Select the branch for the version of MQ to be used: git checkout 9.2.5 or git checkout master
  • For production image download last MQ release tar file from ibm support or ppa (Select something like IBM MQ 9.3 Long Term Support Release for Containers for Linux on x86 64-bit Multilingual), then copy the tar.gz file to mq-container/downloads folder
  • For developer image no need to download it, go to developerwork site to get the image

  • Edit the install-mq.sh file and change the following variable to use AMQP channel

    export genmqpkg_incamqp=1

  • Set up AMQP authority, channel, and service properties by adding the contents of the add-dev.mqsc.tpl file to the bottom of the /incubating/mqadvanced-server-dev/10-dev.mqsc.tpl file in your cloned repository

  • Start the build

export MQ_ARCHIVE_DEV=IBM_MQ_9.3_LIN_X86-64_NOINST.tar.gz 
export MQ_VERSION=9.3 
export LTS=true 
export DOCKER_BUILDKIT=0 
export REGISTRY_USER=jbcodeforce  
export REGISTRY_PASS='quotethepasswordandescape$with\$'
# Development image
make build-devserver
# Production image
make build-advancedserver

If you get this error: Error response from daemon: network with name build already exists do

docker stop build-server 
docker network rm build

Error This is due that docker build does not support--network` anymore.

Configuring the Queue Manager

It is recommended that you configure MQ in your own custom image by adding your own MQSC file into the /etc/mqm directory on the image. This file will be run when your queue manager is created. See this tech note.

Another way is to use mqsc inside of the running container:

docker exec -ti ibmmq bash
dspmq

Compendium

With links to supporting programming languages. * MQ Cheat sheet * High availability * IBM MQ samples and patterns best source to get you jump straight in and play * AsyncAPI MQ Bindings * Administering a queue manager using IBM MQ Console - IBM Cloud * First demo on docker * Develop a JMS point to point application The code of this IBM tutorial is also in this repository under the democlient/MQJMSClient folder so we can test the configuration.

Coding and personal studies

MQ messaging coding challenge

See the MQ Challenges in java note and the Java-studies/mqChallenges.

Some comments:

  • When a publisher publishes a message to a topic string, one or more subscribers for that topic string receives the message
  • A JMS application can use the JMS destination object which maps to a topic in the same way as it would use the destination to map to a queue, in a point to point scenario. For the publication to reach the subscriber successfully, both the publisher and the subscriber must match same topic string. The subscriber will get publications only from the time they subscribe to a topic.
  • If a publication is sent before the subscription by a specific application is created, that application will not get it.
  • Request response or request reply is an integration or messaging pattern where the application that sends a message to another application, requires a reply of some sort from the receiving application.
  • This is often based on the point to point messaging style and can be synchronous (the sending application waits for the response before it times out) and asynchronous (also called request/callback, where the sending application disconnects but sets up a callback to handle a reply).
  • The sending application usually sets a reply-to-destination and a correlation-id so that a response can get back to the right sender application.
  • For the event booking service the reply-to destination has been defined administratively on the queue manager. However, the requester could dynamically create a temporary destination from its JMS session to complete the exchange.

JMS topic subscription code is in TicketSubscriber.java

Code repositories