Retrying paid AI jobs in AFC: what is safe, and what still costs

A single AFC job can wait several minutes on a paid video model. AFC is the generation service I designed and built for an AI film studio’s apps: a Cloudflare Workers API, durable Cloudflare Workflows as the control plane, Go executor containers that do the work, Postgres through Hyperdrive and Drizzle as the record, and R2 for media.

Every layer of that stack retries something: callers retry a POST that timed out, Workflow steps retry a failed call to the container, the executor retries provider calls, and queues redeliver webhooks. I didn’t try to remove those retries, because each one is the right local decision.

The goal was narrower than “every retry is free”. Once a result is stored in R2 under its job, any retry from any layer should find it and read it, not pay for it again. Before that point there is a window where paid work can be repeated. It is a known, bounded window. The design narrows it but doesn’t close it, and the sections below say where it is.

The key and the body hash

Every generation request must carry an idempotency key. On its own a key only answers “have I seen this before?”, which is not enough. A caller with a bug can reuse a key for a different request, and silently returning the old job would hand them the wrong film. So the API also stores a SHA-256 hash of the validated request body next to the key.

The route, paraphrased:

const existing = await jobByIdempotencyKey(key);
if (existing && existing.requestHash !== hash) return conflict(); // 409

const jobId = existing?.id ?? crypto.randomUUID();
const inserted = existing ? false : await insertJob({ id: jobId, hash }); // ON CONFLICT DO NOTHING
const current = existing ?? (await jobByIdempotencyKey(key));
if (current.requestHash !== hash) return conflict(); // lost the race to a different body

Same key and same body returns the existing job. Same key and a different body is a 409 with IDEMPOTENCY_CONFLICT. The key has a unique index and the insert does nothing on conflict, so two identical requests racing each other produce one row, and both responses describe that row. A fresh insert answers 201 and a replay answers 202.

That is the guarantee: one key and one body map to one job row. It is not a guarantee of one Workflow. Each request picks its job ID before the insert, and the request that loses the race still starts a Workflow under its own, losing ID rather than the winner’s. That second Workflow finds no job row under its ID and fails at its first step, before it reaches the executor, so it doesn’t pay for anything. It is still a gap, and the fix is small: start the Workflow with the ID of the row that was read back.

A Workflow named by the job

After the row exists, the API starts a durable Workflow whose instance ID is the job ID. If the Worker crashes between inserting the row and starting the Workflow, the caller retries with the same key, gets the same job ID, and starts the same Workflow name. If the Workflow was in fact started the first time, the second start fails because that instance exists, and the API treats that as success. Only when the instance’s status reads unknown does it answer 503, telling the caller to retry with the same key. The API only starts a Workflow for a job that is still queued, so replaying an old key wakes nothing up. The executor container is addressed the same way: its Durable Object is named by the job ID, so every call for a job reaches the same container.

Checkpointed steps

Inside the Workflow, every side effect is its own checkpointed step, from marking the job running to queuing the terminal webhook, and a completed step is never re-run. If the Workflow is evicted after the executor returned, it resumes at “persist the result” with the executor’s response already recorded, not at “call the provider”.

The executor step once had a subtle bug: a step only retries when its callback throws, and the client returned failures instead of throwing them, so nothing retried. Now a retryable failure is thrown, carrying the last response so an exhausted retry reports the real cause.

One execution per job

The executor call is asynchronous. The Workflow step submits the job, gets a 202, and polls the job’s status every two seconds until the response is terminal. Inside the container, the first submit for a job ID creates an execution and runs it in the background, detached from the request that started it. Any later submit for the same job ID gets that same execution back. A status poll, or a repeated submit, answers 202 while it runs and the stored terminal response once it has finished.

So a step retried while its container is alive attaches to the first generation and reads its result.

This replaced an earlier design. The executor used to refuse a second request for an active job with a retryable JOB_ALREADY_RUNNING, and the step had to wait and try again. My first backoff was short, and a long scene burned through every retry while its first attempt was still rendering. The step’s backoff now starts at ninety seconds and doubles, sized against how long a unit runs rather than how long a network blip lasts, and the step has a fifty-minute timeout.

The output key in R2

The execution lives in the container’s memory, so it dies with the process. When a container is replaced mid-job, the next poll gets a 404, and the client submits the job again to the new process. That process has no memory of the job, so the first thing it does, before preparing inputs or calling a provider, is check whether the job’s output key already exists in R2:

if existing := storage.Head(ctx, outputKey); existing != nil {
    if existing.WrittenBy != request.JobID {
        return failure("OUTPUT_KEY_CONFLICT", retryable(false))
    }
    return success(existing) // no provider call, and no attempts
}

Every upload is tagged in object metadata with the job that wrote it and the provider that produced it. If the key exists and this job wrote it, the executor returns the stored object. If another job wrote it, that is a conflict, and the executor refuses to overwrite it.

The diagram’s miss branch is the honest limit of the design. If the process was lost after a provider was called but before the upload landed, nothing durable records that the work happened, and the new process pays for it again. The same is true of any retryable failure once a provider has been called, whether R2_UPLOAD_FAILED after a finished clip or a chain that ran out of attempts: the step goes around again, and if that retry reaches a fresh process, the whole chain runs again. The provider documentation in the repository says it directly: a completed upload is idempotent, a failure before upload can repeat paid work.

The window is bounded by the step’s timeout and its five retries, not by a count of provider calls. It opens when a process dies, or a retryable failure ends the run, after a provider was called and before the output is stored.

Cancellation needed one distinction. A cancelled Go context looks the same whether the API cancelled the job or the container was replaced under it, so the executor remembers API cancellations, and only those stay final.

First terminal state wins

Three writers can end a job: the Workflow with a success, the Workflow with a failure, and the cancel endpoint. Each one is a conditional update that only matches a job that isn’t terminal yet:

update(jobs)
  .set({ status: "succeeded", outputKey, completedAt })
  .where(and(eq(jobs.id, id), inArray(jobs.status, ["queued", "running", "waiting"])))
  .returning({ id });
// zero rows back: someone else finished it first

If no row comes back, the writer skips its event and its attempt rows. It doesn’t decide the webhook. The Workflow re-reads the job after persisting and acts on what it reads: if the job ended cancelled, it stops; otherwise it queues the terminal webhook for the stored state, and the delivery row’s ID decides whether that event was already sent. The cancel endpoint only queues generation.cancelled if the job actually ended cancelled.

Attempts, retries and fallback

Provider retries live in one Go attempt runner. Each model in the chain gets a number of attempts, and the error class decides what happens next:

• Authentication, invalid input and cancellation stop the whole chain at once.
• Rate limits, timeouts and other retryable errors back off and try the same model again.
• A refusal also gets the same model’s remaining attempts first, because some models refuse the same prompt intermittently.
• Once a model’s attempts run out, or a provider is unavailable, the chain moves on only if the policy allows fallback for that reason.

One exception is about cost. Some providers get their own per-attempt window that matches their own prediction limit. When that window expires, the runner only knows its local deadline passed; the provider may already have incurred cost for the full run, so the runner doesn’t retry the same model. A fast failure inside the window keeps its retries.

The runner collects every attempt it makes, failed ones included, with the model, duration, error and whatever usage and cost the provider reported, and returns them with the executor’s response.

Collected is not the same as stored. Attempts are written only by the writer that wins the terminal update, so a writer that loses the race drops the attempts it carried. A job recovered from R2 after process loss comes back with an empty list, so the attempts the lost process paid for are never reported. And only the last executor response reaches the database, so attempts from an earlier run of the step that failed retryably are dropped when a later run succeeds. Each of these is spend the cost record doesn’t see.

Webhooks through a queue

A broken callback receiver must never block or repeat paid work, so the Workflow doesn’t call it. It writes a delivery row and puts a message on a Cloudflare Queue, and a consumer signs and posts it.

The delivery ID is stable: the job ID plus the event name. Inserting the row does nothing on conflict, and both the enqueue and the consumer skip a row already marked delivered. Failures back off exponentially, and a message that runs out of retries lands in a dead-letter queue that keeps the exact envelope for an admin replay.

This is at-least-once delivery. An event that isn’t marked delivered yet can be queued again, and a crash between a successful POST and marking the row delivered sends it twice. Receivers are expected to dedupe on the Webhook-Id header, which is the stable delivery ID.

A transactional outbox for style runs

Style runs can pause for days waiting for a human review, and some of their events wake the run’s Workflow, so losing one would strand a run. Run state, decisions and their events commit in one Postgres transaction. A drainer claims a batch with FOR UPDATE SKIP LOCKED, so two drainers never take the same rows, and pushes each row’s next attempt a minute out in the same transaction, which works as a lease if the drainer dies. Like the queue, this is at least once with a stable event ID.

Cost per attempt, and null is not zero

The cost record follows one rule: never claim more than you know. Each stored attempt has a cost in numeric(12,6), never a float, and a source: quoted by the provider, derived from reported units with a rate card, or local. A provider call nobody could price stays null, and null is counted as unpriced everywhere spend is shown.

A step retried on transport failure can report the same job, provider and attempt number twice, and the second report may be the one carrying cost and usage. So the attempt write is an upsert on that triple: it overwrites the telemetry and leaves the attempt’s outcome, error and timing alone. A replay can’t double-count a row.

What null can’t express is an attempt that was never stored. The attempts lost with a process, or dropped with a losing writer, are absent rather than unpriced.

What I’d keep

None of these guards is clever on its own. What made them work was asking the same question at every layer: if this exact call happens twice, what does the second one see? For finished work the answers are things you can read back: a unique key and a hash, an instance name, an execution keyed by job, an object that already exists, a WHERE clause on the current status, and a row that says “delivered”.

Asked honestly, the same question shows where the answer runs out. Before the upload lands, the second call sees nothing, and it pays. I’d rather know where that window is, and keep it short, than claim it isn’t there.