Definition :
Sonic MQ implements a mechanism to prevent message loss in the event of a queue or topic database reaching capacity. This mechanism is known as "Flow Control".
When It Occurs :
When the system is running normally flow control has no effect; the rate of consumption of messages by services is greater than the rate of production by clients. In this case no messages build up on queues and everything works smoothly. However, in the scenario where a service cannot keep pace with the message throughput, then the entry queue will fill up with unprocessed messages, and this will ultimately have a ripple effect in the processing chain; ultimately resulting in clients being unable to send new messages.
This ability for the impact to be felt by the client applications is why the knowledge of flow control is so important, which makes suggestions and standards on how to avoid flow control so critical to the success of the platform.
State 1: Normal Flow
This is the optimal state for any message flow solution; when everything is running normally flow control has no effect. The rate of consumption of messages by services is greater than the rate of production by clients. In this case no messages build up on queues and everything works smoothly.
State 2: Queue Fills
There are numerous reasons why a service's processing rate can degrade;
• Inaccurate assessment of original throughput capability of service
• Dependency not meeting its agreed SLAs; database, host system or service dependency
• Various infrastructure or communication issues
In any scenario where the service is not processing, the service entry queue will fill with pending messages. When Sonic becomes aware of a full queue, it will attempt to prevent more messages being added to that queue; localised flow control is activated.
State 3: Control Propagated Backwards
When Sonic has stopped the service entry queue from receiving new messages, it then propagates this flow control back towards the source of the message. Ultimately, this will impact the DRA routing queues that are critical to the movement of messages between the different tiers. The DRA Routing queues are common to all routing traffic and as such, when their ability to accept messages has been denied, the entire flow of messages between the tiers has ground to a halt.
State 4: Flow Control Activated
Again, when the DRA routing queues fill, Sonic will again attempt to move the flow control back towards the source of the problem and will implement flow control on the next layer queues that the clients interface with. Ultimately this will result in clients timing out when they attempt to place new messages on the queues, and basically renders all Business Services unavailable.
Flow Control and Topics :
Do not assume that using Topics instead of Queues means Flow Control will never occur. A major potential cause of flow control is with durable subscribers who've become disconnected. The topic starts to fill up with more and more messages awaiting that subscriber's return. Eventually it can fill the Brokers' Topic Database and underlying filesystem. At that point ALL topics on that Broker are affected and in Flow Control, as all topics share the same underlying Topic Database.
A very similar scenario can be encountered with a normal, but slow, subscriber (i.e. not a Durable Subscription), gradually filling the topic database with the same eventual results.
Flow Control Avoidance
To avoid flow control issues, the following recommendations should be followed:
• Business Service entry queues MUST be correctly sized
• All messages MUST have a time to live set to just over the client connection timeout.
• All Business Services MUST implement Topics Entry Endpoints on the Broker wherever possible instead of Queues.
• The delivery mode of the message MUST be set appropriately, e.g. a delivery mode of BEST_EFFORT the message will be immediately discarded in the event of a service being unavailable
Dead Message Queue :
Messages sent to Queues, where the Time To Live (TTL) has been exceeded before the message is consumed are sent to the Dead Message Queue (DMQ) if the 'preserve undelivered' header is set. This is something to be aware of when setting the header elements of the message. In this scenario, there is a very real possiblity that the DMQ could become full, resulting in the activation of Flow Control.
The TTL expiry does not apply to Topic messages; messages that expire in Topics are never sent to the DMQ, so this further bolsters the argument for implementing Topics rather than Queues where possible.
Sonic MQ implements a mechanism to prevent message loss in the event of a queue or topic database reaching capacity. This mechanism is known as "Flow Control".
When It Occurs :
When the system is running normally flow control has no effect; the rate of consumption of messages by services is greater than the rate of production by clients. In this case no messages build up on queues and everything works smoothly. However, in the scenario where a service cannot keep pace with the message throughput, then the entry queue will fill up with unprocessed messages, and this will ultimately have a ripple effect in the processing chain; ultimately resulting in clients being unable to send new messages.
This ability for the impact to be felt by the client applications is why the knowledge of flow control is so important, which makes suggestions and standards on how to avoid flow control so critical to the success of the platform.
State 1: Normal Flow
This is the optimal state for any message flow solution; when everything is running normally flow control has no effect. The rate of consumption of messages by services is greater than the rate of production by clients. In this case no messages build up on queues and everything works smoothly.
State 2: Queue Fills
There are numerous reasons why a service's processing rate can degrade;
• Inaccurate assessment of original throughput capability of service
• Dependency not meeting its agreed SLAs; database, host system or service dependency
• Various infrastructure or communication issues
In any scenario where the service is not processing, the service entry queue will fill with pending messages. When Sonic becomes aware of a full queue, it will attempt to prevent more messages being added to that queue; localised flow control is activated.
State 3: Control Propagated Backwards
When Sonic has stopped the service entry queue from receiving new messages, it then propagates this flow control back towards the source of the message. Ultimately, this will impact the DRA routing queues that are critical to the movement of messages between the different tiers. The DRA Routing queues are common to all routing traffic and as such, when their ability to accept messages has been denied, the entire flow of messages between the tiers has ground to a halt.
State 4: Flow Control Activated
Again, when the DRA routing queues fill, Sonic will again attempt to move the flow control back towards the source of the problem and will implement flow control on the next layer queues that the clients interface with. Ultimately this will result in clients timing out when they attempt to place new messages on the queues, and basically renders all Business Services unavailable.
Flow Control and Topics :
Do not assume that using Topics instead of Queues means Flow Control will never occur. A major potential cause of flow control is with durable subscribers who've become disconnected. The topic starts to fill up with more and more messages awaiting that subscriber's return. Eventually it can fill the Brokers' Topic Database and underlying filesystem. At that point ALL topics on that Broker are affected and in Flow Control, as all topics share the same underlying Topic Database.
A very similar scenario can be encountered with a normal, but slow, subscriber (i.e. not a Durable Subscription), gradually filling the topic database with the same eventual results.
Flow Control Avoidance
To avoid flow control issues, the following recommendations should be followed:
• Business Service entry queues MUST be correctly sized
• All messages MUST have a time to live set to just over the client connection timeout.
• All Business Services MUST implement Topics Entry Endpoints on the Broker wherever possible instead of Queues.
• The delivery mode of the message MUST be set appropriately, e.g. a delivery mode of BEST_EFFORT the message will be immediately discarded in the event of a service being unavailable
Dead Message Queue :
Messages sent to Queues, where the Time To Live (TTL) has been exceeded before the message is consumed are sent to the Dead Message Queue (DMQ) if the 'preserve undelivered' header is set. This is something to be aware of when setting the header elements of the message. In this scenario, there is a very real possiblity that the DMQ could become full, resulting in the activation of Flow Control.
The TTL expiry does not apply to Topic messages; messages that expire in Topics are never sent to the DMQ, so this further bolsters the argument for implementing Topics rather than Queues where possible.
Comments
Post a Comment