Symptoms, a terminal, and nobody telling you which layer lied.

Processing time exceeds the visibility timeout. When a worker takes longer than 30s, the queue decides the message was never handled and makes it visible again — so a second worker legitimately picks up an order the first worker is still processing. Both complete. Both fulfil.

Nothing is broken. The queue honoured its contract exactly. The bug is that your code assumed “received” meant “nobody else will get this”.

The retry wrapper and the concurrency deploy both correlate with the symptom and neither causes it — which is precisely why the incident took three days in real life.

This challenge deliberately does not have one fixed cause — the environment varies it, so you cannot pattern-match your way out. What transfers is the sequence.

You bisect by layer, not by guess. Does the latency appear in a packet capture or only in the application? Does it survive replacing the hostname with an IP? Does it track pool checkout time? Does the downstream see one request or four? Each question kills whole branches of the candidate list at once.

Round-numbered delays are a strong hint toward retransmission or a fixed timeout somewhere. Eight to twelve seconds of nothing, with no CPU, is almost never “the code is slow”.

The side effect and the offset commit are two separate events, and a deploy is exactly when the gap between them gets exposed. A consumer that performs work and then dies before committing has, from the log's point of view, never done that work. On restart it is handed the same events again.

A rebalance during rolling deployment multiplies the chance of landing in that window, which is why it looks like “deployments cause it” rather than “our commit boundary was always wrong”.

The repair is not a longer timeout or a slower rollout. It is deciding whether your side effects are idempotent — and if they cannot be, moving the commit boundary so that replay is harmless.