Tuesday 21 April 2009

Durable messages and persistent subscriptions

...or is it persistent messages and durable subscriptions? Two words that sometimes cause confusion to users of Open Message Queue and other JMS providers are durable and persistent. In ordinary English they mean much the same thing, but in JMS they have precise and quite distinct meanings.

Persistent


A message can either be persistent or non-persistent. This is known (rather confusingly) as the delivery mode of the message, and is specified when the message is sent by its originating client.

According to the JMS specification, when a message is marked as persistent, the JMS provider must "take extra care to insure the message is not lost in transit due to a JMS provider failure".

What the specification doesn’t say, however, is that persistent messages will always be persisted. We’ll come back to this below.

How do you specify that a message being sent should be persistent or not persistent?

Messages are persistent by default. There are two ways to specify that messages should be non-persistent:

You can specify that the MessageProducer (the QueueSender or TopicProducer) should send persistent messages by calling
messageProducer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
Alternatively you can specify the deliveryMode on a per-message basis at the point where the message is sent:
messageProducer.send(message, deliveryMode, priority, timeToLive);
One word of warning: the Message object has a method setDeliveryMode(). It might seem that you can call this before sending the message to specify whether the message should be persistent or non-persistent. But you can’t – it doesn’t work. This counter-intuitive behaviour is mandated by the JMS specification.

Durable


A subscription on a topic can either be durable or non-durable. The term durable applies to the subscription, not the topic itself or the messages sent to it. And it applies only to subscriptions on topics, not subscriptions on queues.

Let’s review what we mean by subscription.

Here’s an example that creates a non-durable topic subscription and uses it to consume a single message:
TopicConnection conn = connectionFactory.createTopicConnection();
TopicSession sess = conn.createTopicSession(false,Session.AUTO_ACKNOWLEDGE);
TopicSubscriber sub = sess.createSubscriber(topic);
conn.start();

Message mess = sub.receive(1000);

conn.close()
When the connection is closed, the subscription is closed as well (you can also close it explicitly by calling sub.close()).

Here’s an example that creates a durable subscription on the same topic:
TopicConnection conn = connectionFactory.createTopicConnection();
conn.setClientID("myClientID");
TopicSession sess = conn.createTopicSession(false, Session.AUTO_ACKNOWLEDGE);
TopicSubscriber sub = sess.createDurableSubscriber(topic,"mysub");
conn.start();

Message mess = sub.receive(1000);

conn.close()
As you can see, it’s almost identical except that the call to createSubscriber(Topic topic) has changed to createDurableSubscriber(Topic topic, String subname), where subname is a name used to identify the subscription.

In fact whilst the non-durable and the durable subscriptions are open they behave in much the same way. Whenever a new message is sent to the topic, a copy of that message is sent to both the non-durable and the durable subscriber. An open subscription is described as being active.

The difference can be seen when the subscription is closed. When a non-durable subscription is closed it is considered to no longer exist and is deleted from the MQ broker. When a durable subscription is closed, however it continues to exist in the MQ broker, accumulating messages. A closed durable subscription is described as being inactive.

Consider the case where, after the subscription is closed, a new message is sent to the topic.

If the client subsequently connects once more and creates a non-durable subscription, it will not receive the message that was sent whilst the subscription was not in existence. It will only receive new messages that were sent since the new subscription was created.

However if the client connects once more (supplying the same client ID as before) and creates a durable subscription (supplying the same durable subscription name as before) then the durable subscription is re-opened and the message that was published whilst it was closed will now be received.

Note the importance of client ID when using durable subscriptions: a durable subscription is identified by is subscription name combined with the client ID that created it. You can have multiple durable subscriptions with the same name but different client IDs.

You might be wondering why the concept of durable and non-durable subscriptions applies to topics but not to queues. The answer is that you can think of a queue as having a single, built-in durable subscription shared by all consumers.

How Durable and Persistent Interact


I mentioned above that persistent messages aren’t always persisted. We can understand this by considering queues and topics in turn.

Queues


When a persistent message is sent to a queue, the MQ broker saves it on disk in a data structure representing the queue.

If a consumer is connected the message will be dispatched to it. If there are no consumers connected the message will remain saved on disk until a consumer connects, whereupon it will be dispatched.

When a non-persistent message is sent to a queue, the broker saves it in memory. When a consumer connects the message is dispatched to it. If the broker is restarted before the message is sent then it will be lost forever.

Topics


When a persistent message is sent to a topic, the MQ broker’s behaviour depends on what subscriptions (if any) are present and in particular whether these subscriptions are durable or non-durable.

If there are any durable subscriptions in existence, whether active or inactive, then a copy of the message will be saved on disk in data structures representing each durable subscription.

If any of these durable subscribers are active a copy of each message will be dispatched to them. For those durable subscribers that are inactive, the message will remain saved on disk until they become active.

If there are any non-durable subscribers present then the broker will also dispatch a copy of the message to them. However the message is not saved, either on disk or in memory, for the benefit of any non-durable subscribers that appear later. It may be saved in any durable subscriptions that exist, but these are not relevant to non-durable subscribers. A non-durable subscriber which connects after the message was sent will not receive the message.

This is a direct consequence of the semantics of JMS topics: a non-durable subscriber only receives messages that are sent whilst it is active.

Note that whereas persistent messages are saved on disk before being dispatched to an active durable subscriber to avoid message loss in the event of failure, this is not necessary when dispatching to a non-durable subscriber. This is because the JMS specification allows non-durable subscriptions to be less reliable:
...nondurable subscriptions... are by definition unreliable. A JMS provider shutdown or failure will likely cause... the loss of messages held by... nondurable subscriptions. The termination of an application will likely cause the loss of messages held by nondurable subscriptions... (JMS Specification section 4.10)
Now let's consider what happens when a non-persistent message is sent to a topic, first in the case of durable subscriptions and second in the case of non-durable subscriptions.

If there are any durable subscriptions on this topic, then a copy of the message is sent to those durable subscribers that are active. For those durable subscriptions that are inactive, a copy of the message is saved in memory and sent to them when they next become active.

This saved message will be lost if the broker is restarted. Since non-persistent messages are not saved on disk, a broker restart means that any inactive durable subscriptions that have not yet received the message will miss out on the message.

Again, this behaviour is expressly permitted by the JMS specification, which states:
If a NON_PERSISTENT message is delivered to a durable subscription... delivery is not guaranteed if the durable subscription becomes inactive (that is, if it has no current subscriber) or if the JMS provider is shut down and later restarted (JMS specification section 4.10).
If there are any non-durable subscribers present then the broker will also dispatch a copy of the message to them. Obviously the message is not saved on disk. Nor is it saved in memory for their benefit or that of any non-durable subscribers that appear later. So any non-durable subscribers that appear after the message was sent will miss out on the message. The same as with persistent messages, this is a direct consequence of the semantics of JMS topics: a non-durable subscriber only receives messages that are sent whilst it is active.

Summary


So what have we learned?
  • Messages may be identified as persistent or non-persistent. This is specified when the message is first sent.

  • A subscription on a topic may be durable or non-durable. This is specified when the subscription is created.

  • The term durable doesn’t apply to subscriptions on queues, though it sometimes helps to think of a queue as behaving as if it has a single durable subscription shared by all consumers.

  • Persistent messages are persisted when received by the MQ broker, for queues and durable subscriptions but not for non-durable subscriptions.

  • Non-persistent messages are delivered immediately to connected eligible consumers. They are never persisted on disk. In the case of a queue or durable topic subscription they may be saved in memory until a consumer appears, though this is not guaranteed.

No comments: