Persistent Sending Queue

A persistent sending queue stores pending outgoing records on durable storage so supported restarts can resume delivery. It protects only the data accepted and persisted under its storage rules. It does not automatically protect earlier in-memory processing or the original source. It is also one answer to backpressure: buffering to disk absorbs a slow consumer instead of pushing the slowdown back to the producer, which trades one failure mode for another.

Fluent Bit shows the shape of such a queue concretely. Its engine groups emitted records into chunks — “each chunk has an average size of 2 MB” — and the buffering mode decides where a chunk lives. With memory-only buffering the data stays in memory “until it’s ready to process and route”; with filesystem buffering Fluent Bit “stores each chunk of buffered data in the filesystem through mmap(2)” and also keeps an identical copy in memory when there is room. The disk is therefore not a fallback that engages at the limit — in this design it is where the chunk is written in the first place.

The limits are the part worth reading before deployment, because they decide what happens when the destination stays down. Memory buffering has mem_buf_limit, after which “the plugin pauses until more memory becomes available.” Filesystem buffering has storage.max_chunks_up, the maximum number of chunks that may sit in memory at once, and storage.pause_on_chunks_overlimit, which when set to on makes the plugin “stop both memory buffering and filesystem buffering until more memory becomes available.” Pausing the input is the honest behavior — it is where the queue stops absorbing and the slowdown reaches the source, which is the thing a disk buffer was supposed to postpone rather than abolish.

If an upstream batcher acknowledges a record while retaining it only in memory, a crash before enqueue can lose it despite the downstream disk queue. Trace acknowledgment and checkpoint boundaries through the entire pipeline. Queue storage and file-read checkpoints serve different purposes even when they share one storage extension.

Bound disk use and specify full-queue and retry-exhaustion behavior. A lost acknowledgment can still cause replay — so the consumer needs idempotent writes or deduplication — while disk failure, expired retention, or permanent rejection can cause loss. Capacity helps survive an outage only if the source remains recoverable and recovered output capacity can drain the backlog. Where it cannot, the choice is between a growing queue and shedding — made deliberately or made for you when the disk fills. Test actual component versions rather than inferring guarantees from the word persistent.

Further reading

• Fluent Bit Documentation, Buffering and storage

See the worked examples in Log and telemetry collection.