Background jobs need a memory
Moving work into a queue is the beginning. Durable state, safe retries, and honest progress make it dependable.
“We'll do it in the background” sounds like an implementation detail. To the person waiting for a video export, it is a promise. Their request should survive a closed browser, a restarted worker, and a temporarily unavailable dependency. A queue helps move work between processes, but the product still needs a durable account of what happened.
Separate acceptance from completion
Imagine a service that generates image previews. The upload endpoint can accept the request and return a job identifier while processing continues. The job record is what lets a later page load explain whether the request is queued, running, complete, or failed.
Those states should describe actual events. “Complete” should mean the output has been stored and can be accessed, not merely that a worker started an upload. Similarly, a progress bar should not invent a precise percentage if the system can only observe broad stages. “Generating previews” is more honest than a timer that inevitably reaches ninety-nine percent and waits.
Assume a message can arrive again
Amazon SQS standard queues explicitly use at-least-once delivery, which means consumers must tolerate repeated messages. Other queue systems have their own contracts; read the one you actually operate. Do not make the user-visible operation depend on an assumption that each message appears only once.
A worker can use a stable operation identifier and durable state to recognise completed work. That check still needs concurrency protection: two workers can observe the same unfinished state. Consider which resource owns the claim, how long the claim lasts, and how another worker recovers it after a crash. A queue receipt and an application-level completion record solve different problems.
Retry with a reason
A temporary network failure is a sensible retry candidate. A malformed source file usually is not. Classify failures sufficiently to avoid spending the entire retry budget on an input that cannot succeed. Backoff and jitter can spread retries, while a maximum attempt count prevents an unhealthy dependency from receiving endless traffic.
Keep failed jobs inspectable. Record a safe error summary and enough diagnostic context to investigate without retaining unnecessary private input. A manual retry should create a clear new attempt under the same operation history, not erase the evidence of the original failure.
Design recovery before the happy path is forgotten
Walk through a crash after downloading the source, after creating the output, and after storing the output but before acknowledging the message. For each point, ask what the next worker will know. The answer should come from durable records, not from variables that disappeared with the process.
The most useful background system is not the one that never encounters failure. It is the one that can explain its state, resume safely where possible, and tell a person when intervention is genuinely needed.
Further reading: Amazon SQS: At-least-once delivery.