Anypoint Messaging Queue, also recognized as Anypoint MQ, is a cloud-based service for message queuing that facilitates secure and seamless data transfer among multiple independent instances of one or several applications operating within a shared environment. Serving as an intermediary, Anypoint MQ ensures the reliable and secure transmission of data between applications, supporting advanced asynchronous messaging scenarios such as queuing and publishing/subscribing. It provides fully hosted and managed cloud message queues and exchanges. As an integral part of the Anypoint Platform, Anypoint MQ incorporates support for environments, business groups, and role-based access control (RBAC), ensuring smooth customer experiences across channels and dependable integration of devices for IoT applications with enterprise-level functionality. The management of extensive data movement between applications, systems, and services would be considerably more challenging without the presence of an MQ.
A queue functions as a temporary storage space for messages, facilitating their sending or receiving. Essentially, it is an arrangement of data items in a specific order to enable easy retrieval. At Anypoint MQ, queues are established to act as intermediaries between various applications and services. Within the Anypoint platform, users can create different types of queues, including standard queues, FIFO queues, and dead letter queues (DLQ).
*Note: An enterprise subscription on your Anypoint Platform account is required to utilize Anypoint MQ.
Types of Destinations:
- Queues: In a standard queue, there is no predetermined order for message delivery, making it suitable for applications where quick delivery is crucial. Anypoint MQ supports up to 120,000 in-flight messages per queue, with each message delivered at least once.
- FIFO Queues: In a First In First Out (FIFO) queue, messages are delivered in a specific order, prioritizing the sequence of message transmission over speed. Anypoint MQ supports 10 in-flight messages per message group in FIFO queues, ensuring each message is delivered exactly once.
- Exchanges: A message exchange is employed when a user intends to send a message to one or more standard queues bound to the exchange. Users can also create message routing rules to direct specific messages to particular queues.
Dead Letter Queues (DLQ):
The DLQ in Anypoint MQ is akin to any other queue. Messages that fail to be delivered are redirected to the DLQ for later analysis. To designate a DLQ, users must have at least two queues meeting specific criteria.
Client Application:
A Mule app can be registered with the Anypoint MQ connector for messaging queues. After registration, the app generates a “Client App ID” and “Client Secret,” which are utilized in the Anypoint MQ connector configuration in Anypoint Studio.
Operations in MQ:
Anypoint MQ supports five types of operations:
- Publish: Creates and sends an Anypoint MQ message to a specified destination, such as a queue, FIFO queue, or message exchange.
- Consume: Retrieves a single message from any queue at any point in the flow using the acknowledgment strategy.
- Subscriber: Listens for new messages and consumes them as soon as they arrive at the destination.
- ACK (Acknowledgment): Provides manual acknowledgment for messages to the queue.
- NACK (No Acknowledgment): Provides manual non-acknowledgment of messages to the queue.
Acknowledgment Modes:
Anypoint MQ offers three acknowledgment modes:
- Immediate: Acknowledges the message as soon as it is consumed from the queue and before processing by the Mule flow.
- Automatic: Acknowledges a received message only if the flow execution is successful; otherwise, the message is sent back to the queue for redelivery.
- Manual: Allows the app logic to handle message acknowledgment, generating a unique ackToken for each received message.
Practical Implementation – Use Case:
Consider an employee management system where data needs to be managed in two different systems. In this use case, an API is created to read client/employee data from a CSV file and store it in two distinct queues. Subsequently, the data is fetched from the queues using either Subscriber or Consume, and then it is stored in the database.
Sample CSV: [CSV data not provided]
d,firstname,contact,salary,designation 101,Frieda,9146791137,2868526,Marseille 102,Desirae,9334700621,2842950,Dublin 103,Tarra,9911636744,3495827,Riverside 104,Gusella,9490301855,3815923,Funafuti 105,Tarra,9723055450,4045760,Edmonton 106,Ottilie,9257336086,2043371,El Paso 107,Rhea,9907781269,4864561,Ambon 108,Gisela,9033599105,4737095,Amsterdam 109,Cam,9934544276,3640212,Yogyakarta (city) 110,Tonia,9214400081,3382217,Ponta Delgada 111,Tobe,9281274872,3099947,Ponta Delgada 112,Laurene,9867860920,3113101,Córdoba 113,Maisey,9162257400,2896334,Simferopol 114,Jany,9645911254,4170102,Lubumbashi 115,Meg,9694133935,3276687,Miami 116,Kial,9422167776,3854913,Yakutsk 117,Gusty,9294140215,2412079,Chihuahua 118,Lynea,9050643996,1343549,Gdańsk 119,Nananne,9074558421,2245403,Sofia 120,Penelopa,9758776479,3849714,Managua
Step 1: Create queues
1. Go to Anypoint Platform. On the home page, click on MQ
2. Click on Destinations in the left navigation pane and then click on the blue plus (+) icon in the top right corner
3. There you’ll get three options: Queue, Exchange, and FIFO Queue
4. Select Queue and fill in all the necessary information.
5. Similarly, create another queue for storing the rest of the data that won’t be stored in the first queue.
Step 2: Create an Exchange
1. Create an Exchange and bind both the queues to the exchange
2. After it is created, open the exchange and routing rules to the queues to set rules for routing the data in the queues. You can do that by “Add Rule” and then fill all the necessary details.
Step 3: Create a Client App
1. Go to Anypoint MQ and click on Client Apps in the navigation pane on the left side.
2. Click on the blue plus (+) icon and enter the name for your client app and click on Save Changes
3. A “Client App ID” and “Client Secret” will be generated that’ll be used to configure Anypoint MQ for Anypoint Studio
Step 4: Configure Anypoint MQ connector
- Create a sample Mule project in Anypoint Studio
- In the “Module Palette”, go to “Search in Exchange” and search “Anypoint MQ”. Add the module for the Anypoint MQ connector and in the “Global Elements”, add “Anypoint MQ Configuration”
- In the configuration, add the “Client App ID” and “Client Secret” that were generated in step 3 and click on ok
Step 5: Read the CSV file and add it to exchange
- Create a flow as shown below
2. In the “Read” connector, select the path where the CSV file is located
3. Then convert the payload to JSON using “transform Message”
4. To send the message to the queue, use the “Publish” connector with the config given below.
5. Wrap the Publish in “For-each” to iterate through the multiple records and send the data to queues
6. Also, add the parameter in the “User Properties” that was defined in the exchange to separate the messages going to the queues
Step 6: Check messages in the exchange
- Go to Anypoint MQ in Anypoint Platform and open the queue you want to check for messages
- Select “Message Browser” from the navigation pane
- Select the number of messages to fetch and the time for which the queue will fetch messages
You’ll see that the messages are now in the queue waiting to be delivered
Step 7: Create DLQ
- Follow the same as creating a queueIn the “Create Queue” panel, toggle the “Assign a Dead Letter Queue” option
3. Select the queue for which you want to create a DLQ and click on create
Step 8: Fetch messages from the queue and send it to the database (Using Consume)
- Create a flow as shown below
2. “Consume” is used to consume the message in the queue and after consumption the message weights for the flow to complete and then it is acknowledged.
3. Setting “Acknowledgement Mode” mode to MANUAL allows you to acknowledge messages at will
4. After the message is consumed, map it with the fields in the database
5. Using the “Insert” connector, the data is inserted into the database
6. “ACK” connector acknowledges the messages if it is successfully inserted into the database and the message is deleted from the queue
If an error occurs while inserting the data in the database, it will go to error handling where the “NACK” connector will give negative acknowledgement to the message and the message will be sent to the DLQ.
Step 8: Fetch messages from the queue and send it to the database (Using Subscriber)
- Create a flow as shown below
2. Adding “Subscriber” instead of listener automatically consumes the messages as soon as they arrive in the queue. Set the acknowledgement mode as MANUAL.
Utilize the “Transform Message” to align the payload with the fields in the database.- Employ the “Insert” connector to insert the data into the database.
- The steps for ACK and NACK are analogous to the aforementioned step 8.
In summary, MuleSoft Anypoint MQ stands out as a robust messaging platform, facilitating smooth communication among applications, systems, and devices. It delivers dependable and scalable message transmission, offers adaptable message routing, and enables seamless integration.
Explore further insights from Caelius Consulting’s Integration Team in the Resource Centre.