Skip to content

Engineering operations · Essay

Stop restarting failed coding agents from the beginning

A retry button is an incomplete recovery strategy. Before expanding autonomous coding, require workflows to preserve completed work and prove which actions can safely run again.

By Nythrex team4 min read

A hand-operated railway switch with a lever and signal lamp beside the rails on gravel.
A hand-operated railway switch beside the tracks. Photo: PIVISO via Wikimedia Commons, CC0. · https://commons.wikimedia.org/wiki/File:Hand-operated_railroad_switch.jpg (CC0)

Imagine a coding agent that has prepared a patch, started its checks and opened a pull request. Its worker then loses contact with the control service. The dashboard shows a failed run, and the easiest response is to start again. But the failed connection tells the operator little about which actions actually finished.

For work that crosses process restarts and external systems, restarting everything is a poor default. My argument is that engineering leaders should require a recovery contract before scaling unattended coding: preserve verified progress, identify external actions and reconcile uncertain outcomes before repeating them. Otherwise, the retry can become another source of work.

The strongest case for starting over

A fresh run is simple to understand. It starts from the current repository, avoids carrying forward a confused conversation and needs less orchestration machinery. For a short, read-only task with disposable output, this can be a sensible tradeoff. Building a workflow platform around every prompt would introduce its own maintenance burden.

The opposing view also has a practical attraction: improve the model and the prompt first, then handle occasional failures manually. That approach deserves a fair trial when interruptions are cheap to resolve. The boundary changes when a run creates durable artifacts, consumes substantial compute or waits for people. At that point, ask what a fresh start will repeat.

The action can finish after the connection fails

In its October 22, 2026 engineering report, Cortex describes a repository clone that completed while a VM snapshot disrupted the connection carrying its result. Its solution combined deterministic execution identifiers with a journal of output and exit status. A repeated request could attach to the existing command instead of launching another. The important detail is the gap between execution and acknowledgement.

Cortex also describes checkpointing workflow steps and parking workflows while they wait, without retaining a running VM. This is an implementation account, not an independent productivity study. It supports a design pattern; it does not establish a savings rate for another team.

The recovery tooling is evolving. In its October 29, 2026 release report, DBOS describes rewinding a workflow to a selected step while preserving its identity. Earlier checkpoints return recorded results; the selected step and later steps execute again. It also describes ownership tokens that prevent an obsolete executor from writing further checkpoints after another executor takes over.

Those mechanisms address workflow state. They do not, by themselves, prove that an external action is safe to repeat. A command may already have changed a remote system before its completion was recorded. AWS's 2023 Well-Architected guidance identifies retries of non-idempotent calls as a cause of duplicate results and recommends bounded retries with backoff and jitter.

Verify that services are idempotent before implementing retries.

AWS Well-Architected Framework, 2023 guidance

Applied to coding agents, the management consequence is straightforward: a green run after a restart is incomplete evidence of efficient recovery. In the hypothetical workflow above, inspect whether the same patch was regenerated, checks were repeated unnecessarily or another pull request was created. The bill can include model usage, compute and human reconciliation. Treat their scale as a measurement question, rather than assuming either a major loss or a negligible cost.

Rows of cooling units on the roof of a large data centre, seen from above.
Cooling equipment on a data centre roof. Photo: Rsparks3 via Wikimedia Commons, CC0. · https://commons.wikimedia.org/wiki/File:Data_center_roof.jpg (CC0)

Make recovery an acceptance test

The following is a proposed operating standard, not a claim that every team needs a particular workflow engine. Apply it to the persistent automations whose interrupted work would be expensive or difficult to reconstruct.

  1. Define what survives. Record the task identity, repository revision, workflow definition, artifact references and completed steps outside the worker's memory. Separate orchestration checkpoints from the agent's conversation history. Decide which outputs remain valid after a new commit or changed requirement; recovery must preserve useful work without silently trusting stale work.
  2. Give consequential actions stable identities. Document how the system recognizes an existing command, pull request or provisioned environment. Specify the downstream service's deduplication guarantees and their limits. Where it cannot safely establish the outcome, require reconciliation or operator review before repeating the action. A new attempt should not silently create a new logical operation.
  3. Make waiting explicit. Require a design for releasing compute while awaiting a review, approval or external event, with enough persisted state to resume. Compare that design with the operational cost of keeping an environment ready. Include restoration and storage in the comparison rather than counting only the time the model is active.
  4. Exercise the uncertain boundary. In an isolated test environment, interrupt a worker after an external action succeeds but before its acknowledgement is stored. Repeat an incoming event and restart the coordinator. Require evidence of which work resumed, which work executed again and why. Also test cancellation and a deployment that changes the workflow definition.
  5. Put recovery on the operating report. Track interrupted runs, recovery duration, repeated model calls, repeated compute and manual intervention. Separate the cost of the original task from recovery overhead. Keep usage valuation distinct from invoiced charges or prepaid credits, and retain unknown amounts as unknown. Expand concurrency only when the recovery behavior is explainable.

The Nythrex view: ask for recoverable work

Our editorial recommendation is to bring the interrupted-run demonstration into the management review alongside the successful demo. Ask the workflow owner to show the last confirmed result, the unresolved action and the rule governing the next attempt. People remain responsible for deciding whether preserved work is still valid and whether uncertain effects need intervention. That makes recovery a reviewable engineering decision.


Sources

Frequently asked questions

Is it safe to retry a failed coding agent?

It depends on which actions completed and whether repeating them can create additional effects. AWS's retry guidance recommends checking idempotency and limiting retries; use persisted progress and reconciliation to establish what the agent should do next.

Does a durable workflow prevent duplicate external actions?

Checkpointing records progress, but external actions still need an appropriate deduplication or reconciliation mechanism. Cortex's engineering report illustrates this with execution identifiers and command journaling inside its agent environments.

    Stop guessing.
    Start seeing.

    No card and no subscription: paid plans have not launched yet.

    Create accountNo card, no subscription