Background Jobs That Survive a Restart — A Field Guide

A user submits a request—generate a report, send a confirmation email, process a payment webhook. Your app accepts it, hands it to a background worker, and returns a cheerful "we're on it." Two minutes later your cloud host idles the server down. The worker never finishes. No error surfaces. No alert fires. The user just waits, then gives up, then leaves a one-star review that says your app "doesn't work."

That silence is the subject of this guide.

Why does my app lose work every time it restarts or redeployment happens?

Vibe-coded MVPs almost always store background jobs in the running process's memory—a list, a queue object, a variable that lives inside the server itself. When that server restarts (on a new deploy, after a crash, or because a cloud host recycled an idle container), that memory is gone. The jobs were never written anywhere durable. This is a structural gap in how the prototype was assembled, not a mistake in your logic.

The pattern is close to universal in AI-generated MVPs. The AI tools that helped you build fast made a reasonable trade-off: keep the demo working, skip the infrastructure scaffolding that only matters under real load. A demo runs on a laptop or a single stable server. No restarts, no idle timeouts, no concurrent users hammering the same endpoint. Everything looks clean.

Real production is different. Cloud platforms restart containers on every deployment—sometimes more than once. Hosts idle down free-tier or low-traffic servers after a period of inactivity. A sudden spike in traffic can crash a worker process. In each case, any job that existed only inside the running process simply ceases to exist.

The reader-facing consequence: your app behaves correctly in your demo and incorrectly for your first real users, with no obvious clue as to why.

What actually breaks for my users when a background job dies mid-run?

Emails go unsent, payments stay unconfirmed, and reports never arrive—because the task that was supposed to produce those outcomes died quietly before it finished. Users see none of that. They see an app that promised to do something and then didn't.

Work through a few realistic scenarios. A user signs up and expects a welcome email with their account details. The worker that sends it restarts mid-flight. The email never arrives. The user assumes the signup failed and tries again—or assumes your product is broken and moves on. A payment webhook arrives from a processor, your app queues a job to update the order status, and the server restarts before the job runs. The payment is captured but the order is never fulfilled. Now you have a billing dispute and a confused customer.

Reports are another common casualty. A founder demo-ing to an investor schedules a data export. The export job sits in memory. A deploy happens between scheduling and execution. The export never runs. The founder refreshes the screen in front of the investor.

None of these failures produce an error the user can report. They produce silence—and silence reads as unreliability. Before a pilot or an investor demo, "silently broken" is exactly the category you cannot afford to be in. The fix is not cosmetic. It is foundational.

How do I make sure queued work survives a restart without rebuilding everything?

Write each job to a durable store—a database row, a file, or a dedicated message broker—before any execution begins. If the job exists outside the running process, a restart cannot erase it. This is a persistent queue, and adding one is an addition to your existing code, not a replacement of it.

The mental model is straightforward. Right now, your app probably does this: receive a request, create a job object in memory, hand it to a worker function. Change one step: before handing the job to the worker, write a record of it somewhere that survives a process restart. A simple database table works. Each row represents one pending job—its type, its parameters, its status (pending, running, complete, failed), and a timestamp.

When your worker starts, it reads from that table rather than from an in-memory list. When it picks up a job, it marks the row as "running." When it finishes, it marks it "complete." If the process restarts mid-run, the row is still sitting there with status "running" or "pending." A restart routine can find those rows and re-queue them.

This approach fits around what you already built. Your existing business logic—the code that sends the email, processes the payment, generates the report—does not change. Only the layer that hands work to that logic changes. That boundary is where the durability lives.

How do I stop the same job from running twice if the worker restarts mid-task?

Design each job so that running it a second time produces exactly the same outcome as running it once. That property is called idempotency, and the practical technique is simple: before your worker acts, check whether the action has already been completed, using a status flag or a unique key stored with the job record.

Here is why this matters. A persistent queue solves the "lost job" problem. But it introduces a new risk: a job that was halfway through when the worker crashed will be retried. If your worker sends an email, processes a charge, or writes a record, and it retries a half-finished job, you can double-charge a customer, double-send a notification, or create duplicate records.

The fix is a guard at the start of each job. Before sending the email, check whether the job's status is already "sent." Before processing a charge, check whether a transaction ID has already been recorded against that job. Before writing the report row, check whether it already exists with a matching unique key. If the answer is yes, the worker exits cleanly without repeating the action.

You do not need to redesign the entire job. Add one conditional check at the top of each worker function: read the current status from the durable store, return early if the action is already done, otherwise proceed and record completion immediately after. That check is the difference between a harmless retry and a double charge.

How do I know my background jobs are actually finishing—and get alerted when they are not?

A watchdog process—a system service, a container restart policy, or a lightweight supervisor—detects when a worker has stopped and restarts it automatically. Paired with a simple heartbeat log your app writes on every successful job completion, that combination tells you whether work is actually finishing without requiring an engineering degree to read.

Silence is not success. This is the rule that the in-memory queue pattern violates most damagingly, and it is the rule that process supervision restores. If your worker crashes and nothing is watching it, jobs pile up in the queue unprocessed. No alert fires. You find out when a user complains—or when an investor asks why the report they requested an hour ago still says "pending."

A watchdog is not a complex system. Most cloud platforms let you declare a restart policy on a container: if the process exits, restart it. That single configuration change means a crashed worker comes back within seconds rather than sitting dead until you notice. On a traditional server, a lightweight supervisor process watches the worker and does the same.

The heartbeat is the other half. Every time a job completes successfully, write a row: job type, completion time, result. Keep a week of that log. Now you can answer the question "are my background jobs working?" by looking at a table rather than guessing. If the last completion timestamp for a given job type is two hours old and you know jobs are generated every fifteen minutes, something is wrong—and you know before your user does.

Set a simple check: if no heartbeat has been written for a job type in twice its expected interval, send yourself an alert. That threshold keeps the noise low while catching real failures. You do not need a dedicated monitoring platform to do this. A scheduled database query and an email to your own inbox is enough to start.

Three patterns—persistent queues, idempotent workers, process supervision—close the gap that vibe-coded prototyping leaves open. None of them require discarding what you built. Each one is an addition to an existing layer: where the job is stored, how it handles a retry, who notices when it stops. Together, they make the silence your users currently experience into something your app can explain, recover from, and report on.

The next step is the simplest one. Open your codebase and find where background jobs are created. Check whether that creation writes anything to a database before handing work to a worker. If the answer is no—that is where to start.