5 useful message queue patterns

I love message queues and find them incredibly useful. This article outlines several cases where introducing a queue can make your system simpler and more robust.

Messaging is a vast topic, but here are a few patterns that I've found most useful on real world projects.

Task Queues

By far the most common queueing pattern. Producers write messages to queues and don't wait for a reply. Consumers listen to the queue and process messages.

Useful for: * Performing work in the background that users don't need immediately * Transparently scaling work (just add more consumers)

Common implementation semantics: * Consumers receive messages in FIFO order * Messages are redelivered if the consumer doesn't explicitly ack or delete them

Delayed Jobs

Typically a minor variant on the job queue. Producers can specify that the message delivery be delayed.

Useful for: * Retrying integration with a remote system. Example: You write a job to bill customers through PayPal at the end of each month. If paypal is unreachable you requeue the task with a 24 hour delay. * Deferred state cleanup. You allocate a resource and queue a delayed message that tells a consumer to cleanup the state in a few hours. * Grace periods. Example: You want a customer to be able to cancel sending an email or placing an order for up to x minutes. You enqueue the job with a delay. If the customer cancels within the grace period you simply delete the message from the queue.

Fanout

Fanout destinations are similar to email aliases. The destination that producers write messages to is an alias that maps to one or more child queues. When the broker receives a new message it delivers that message to each child queue. Consumers dequeue from the child queues, not from the fanout destination.

With an email alias the sender doesn't need to know who is subscribed to the alias. The composition of the alias can change at any time.
Similarly a fanout queue acts as a logical destination. The child queues that the fanout queue points to can change at any time without impacting the producers.

Useful for: * Event notification. When an event occurs the producer sends a message. Subsystems that require notification are added to the fanout configuration. Each receive a copy of the message. * Logging

Message Groups

Imagine you have a system that accepts data from a 3rd party as a stream of messages. Each message represents a different write operation and is related to a single entity in your system. Each entity is independent.

So far this looks like a fine problem for a job queue. Messages come in and go into a single job queue. Consumers dequeue messages in FIFO order and process them.

But this doesn't work due to race conditions. If two consumers each dequeue a message related to the same entity and apply them in parallel then state could be modified out of order.

You could write messages to separate job queues -- one per entity. But then consumers would have to have some way of subscribing to all the queues. As the number of entities grows, this could get complicated.

This is the problem that message groups solve. The solution works like this:

• Producers set a message group id header on each message. The details are implementation dependant. But in our example, the message group id would be something like: [entity type]:[entity id]
• All messages are sent to a single destination
• Consumers subscribe to the single destination, as though it were a single job queue
• The queue groups messsages by message group id and guarantees that only a single consumer is sent messages for each group id at any given time.

You can also use this pattern to ensure fair resource utilization across multiple customers when setting up a job queue.

• Customers enqueue background jobs. You set the group id to the customer id.
• This ensures that each customer gets a single consumer, but no more

I first learned about this pattern when reading the ActiveMQ Message Group docs.

RPC

Queues are usually used to route messages asynchronously. However, using return queues you can turn a regular job queue into a load balanced synchronous remote service.

Here's how it works:

• Producers set a "Reply-To" header that specifies the name of the queue to send the reply message to. This is typically a single use UUID. Some implementations hide this detail from you. Messages are sent to the job queue normally.
• Producer blocks on the Reply-To queue, waiting for a response message
• Consumers dequeue normally and process the message
• Consumer sees the "Reply-To" header and sends a response to that queue
• Producer reads response

This pattern allows the same consumer code to function asynchronously or synchronously based simply on the presence of the Reply-To header on the inbound message.

Of course the message queue can become a potential bottleneck, and only queues with very low latency are appropriate for this pattern. You probably wouldn't want to attempt this with a hosted service like SQS.

---

General Purpose Queues

name	model	wire protocol	FIFO	task queues	delayed jobs	fanout	message groups	rpc	notes
Amazon SQS/SNS	SaaS	HTTP	No	Yes	No	Yes (SNS to SQS)	No	Not recommended due to latency	Pros: Nothing to install. Good for integrating remote systems. Cons: High latency. Messages may be delivered more than once. Not FIFO
RabbitMQ	Open source (erlang)	AMQP or STOMP	Yes	Yes	?	Yes	No	Yes, using temporary destinations	Full featured message broker, but more complex to install than some solutions
beanstalkd	Open source (C)	Custom	Yes	Yes	Yes	No	No	Yes, but Reply-To must be part of your message payload	Very simple and fast. Great choice if you only need job queues.
Kestrel	Open source (Scala)	Custom	No if clustered	Yes	No	No	No	Yes, but Reply-To must be part of your message payload	Used at Twitter. Message queues are transparently partitioned. Designed for high volume.
Gearman	Open source (C)	Custom	Yes	Yes	Not natively	No	No	Yes	Client bindings automatically route jobs to functions on worker processes, simplifying writing consumer code.
ActiveMQ	Open source (Java)	Custom, STOMP	Yes	Yes	Yes	Yes - virtual topics	Yes	JMS client (Java) supports all features. STOMP support is not as robust. In my tests message persistence was not sufficiently reliable.
HornetQ	Open source (Java)	Custom, STOMP	Yes	Yes	Yes	Yes, using diverts	Yes	JMS only	Formerly JBoss MQ. Full JMS implementation. STOMP support is not as robust.

Language Specific

The solutions below offer higher level queueing features and tight integration with a single programming language. Typically this means that your producers and consumers have to use the same language, and in some cases, have to share the same code tree due to how messages are marshalled and consumed.

In addition these solutions typically include monitoring and admin features, which makes it easier to visualize the state of your system.

My notes are a bit incomplete on these systems. They all seem to be optimized for the task queue use case, but please comment with any corrections.

name	language	task queues	delayed jobs	fanout	message groups	rpc	backends	notes
celery	Python	Yes	Yes	No	No	Yes	RabbitMQ (preferred), Redis, beanstalkd, MongoDB, CouchDB, SQL databases	Very active community
resque	Ruby	Yes	?	No?	No?	No?	Redis	Github sponsored. Good visualization tools for monitoring workers/jobs.
octobot	Java, Scala, Ruby, Python	Yes	No?	No?	No?	No?	AMQP (RabbitMQ), beanstalkd, Redis	Very limited docs. Used at Urban Airship

---