Delayed message delivery in RabbitMQ

UPD June 01, 2015: there is a plugin for this now.

A lot of developers use RabbitMQ message broker. It is quite mature but still lacks for some features that one may need. One of them is delayed message delivery: there is no way to send a message that will be delivered after a specified delay (it’s a limitation of AMQP protocol). Hopefully, there is a hack for this.

RabbitMQ logo

Let’s start from dead letters. A message can become “dead” by several reasons, such as rejection or TTL (time to live) expiration. RabbitMQ can deal with such messages by redirecting them to a particular exchange and routing key. We can use this ability to implement delayed delivery. We will create a special queue for holding delayed messages. This queue will not have any subscribers in order for messages to expire. After the expiration, messages will be passed to a destination exchange and routing key, just as planned.


rabbitmq_delayed_1

There is a difficulty: the message TTL expiration mechanism works only with the message at the end of a queue, so all messages behind the last message will wait at least as long as the last. However, this can be solved by declaring a separate queue per delay duration.

rabbitmq_delayed_2

A destination exchange and routing key are set as parameters of a hold queue. So, it is necessary to create a separate hold queue for each exchange-routing key as well.

Now, you see the problem: a lot of temporary queues (one per duration + exchange/routing key). There is a way to get rid of them. Not only messages have the concept of TTL, but queues also. The TTL can be set for a queue, and the queue will be deleted when it is unused for this time (disregarding the presence of messages in it). This “using” includes message consuming and queue redeclaration. Hold queues have no consumers, but we can redeclare them on each message posting. If we specify the queue TTL greater than the message TTL, the queue will surely outlive all messages put in it.

We will need several queue arguments, which will be set on its declaration:

  • x-message-ttl – the number of milliseconds for a message to stay consumed in a queue.
  • x-dead-letter-exchange and x-dead-letter-routing-key – an exchange and routing key where message will be send after the expiration.
  • x-expires – the number of milliseconds for a queue to be unused before it will be deleted.

Talk is cheap. Show me the code.
– L. Torvalds

Let’s come to the code.

As you can see, quite simple.

Dear readers, do you use RabbitMQ or other message brokers? What is your use cases. Share, please.