Set RabbitMQ memory and disk limits

The fix/etc/rabbitmq/rabbitmq.conf

vm_memory_high_watermark.relative = 0.6
disk_free_limit.absolute = 2GB

# Bound queues so one runaway can't consume everything:

rabbitmqctl set_policy max-len ".*" '{"max-length":100000,"overflow":"reject-publish"}' --apply-to queues

Why it matters

A queue with no consumer, or a consumer slower than its publishers, grows without bound. RabbitMQ holds messages in memory (and pages to disk), so an unbounded queue eventually exhausts one or the other — and when it does, RabbitMQ blocks all publishers across the whole broker to protect itself.

That’s the failure mode worth understanding: it’s not one queue that stops, it’s every publisher on the broker that gets flow-controlled to a halt, because the node hit its memory or disk limit. One misbehaving application takes down messaging for all of them.

This closes no security exposure, which is why it’s low. What it prevents is a single overload — accidental or deliberate — becoming a broker-wide outage.

The two watermarks

RabbitMQ has built-in alarms that block publishing before the host actually runs out:

• vm_memory_high_watermark — when the node’s memory use crosses this fraction of system RAM (0.6 = 60%), publishers are blocked until it drops. Default is 0.6; the risk is a host where RabbitMQ shares RAM with other services and 60% of total is more than RabbitMQ should use. Set it to what RabbitMQ should actually be allowed.
• disk_free_limit — when free disk falls below this, publishing blocks. The default is tied to RAM size and is often too low; a broker that pages a large backlog to disk can fill the volume, and a full disk is a hard stop. 2GB absolute is a safer floor on most hosts.

These are safety valves, not tuning knobs. They turn “the host ran out of memory and the node crashed” into “publishing paused, consumers caught up, publishing resumed.”

Bound the queues themselves

The watermarks protect the node; queue length limits protect against the queue that shouldn’t have grown in the first place:

rabbitmqctl set_policy max-len "^orders\." \
  '{"max-length":100000,"overflow":"reject-publish"}' --apply-to queues

overflow decides what happens at the limit:

• reject-publish — refuse new messages, so the publisher learns immediately that something is wrong. Usually what you want: fail loud.
• drop-head — silently discard the oldest messages to make room. Right for telemetry where the newest data matters and old data is worthless; wrong for anything where losing a message is a problem.

A message TTL is the other bound — messages older than a limit expire — which suits queues where stale work is pointless.

Lazy queues and quorum queues

Modern RabbitMQ (3.12+) makes queues store messages on disk by default rather than holding them in memory, which changes the memory picture substantially — a large backlog is far less likely to hit the memory watermark than it was on older versions. Quorum queues (the recommended replicated queue type in 4.x) are disk-based and more predictable under load. If you’re designing new, quorum queues plus a max-length are a more robust combination than classic queues with tuned watermarks.

Verify it worked

Check the current limits and whether any alarm is active:

rabbitmq-diagnostics status | grep -iE 'memory|disk|alarm'
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

An active alarm here means publishing is currently blocked — that’s an incident, not a config check.

See the watermark values:

rabbitmqctl environment | grep -iE 'vm_memory_high_watermark|disk_free_limit'

Find unbounded queues — the ones without a length limit:

rabbitmqctl list_queues name messages arguments --quiet | sort -k2 -n -r | head

A queue with a large and growing messages count and no max-length argument is the risk. Check which queues have a policy applied:

rabbitmqctl list_queues name policy --quiet | grep -v '\s$'

Queues with an empty policy column are running with no length bound.

Before you apply this

reject-publish makes publishers fail when a queue is full — which is the intended behaviour, but it is a behaviour change, and an application that doesn’t handle a publish rejection may crash or lose work rather than back off. Confirm your publishers handle a basic.nack/publish failure before applying reject-publish to a busy queue; if they don’t, drop-head or a TTL may be safer while you fix them.

Lowering the memory watermark can trigger an immediate block. If the node is already using more than the new fraction, setting a lower vm_memory_high_watermark blocks publishing the instant it’s applied. Check current usage first:

rabbitmq-diagnostics memory_breakdown

Set the watermark above current usage with headroom, then let it come down.

disk_free_limit too high blocks publishing on a healthy broker. If you set disk_free_limit.absolute to more free space than the volume ever has, the node is permanently in the disk alarm and nothing can publish. Size it to a real floor (a couple of GB), not a large fraction of the disk.

max-length with drop-head silently loses messages. That’s correct for telemetry and catastrophic for orders. The overflow choice is a per-queue decision about whether losing a message is acceptable — make it deliberately, per queue, not with one blanket policy.

Related

• Scope vhosts and permissions — limits which apps can flood a queue.
• RabbitMQ exposed to the internet — an open broker is an easy flood target.