The Hard Part of Long-Running Agents Is Not the Model

I have spent the last few months building NoodleTomato, an agent system for AI video generation. It takes a brief and works on it for hours: planning a video, delegating to workers, running code, generating scenes and other media, spending real money at nearly every step, and finally delivering something a person is meant to use as-is.

I went in believing the model would be the limiting factor. It was not. The model was fine. What kept breaking was everything around it.

For example, a typical NoodleTomato job might be a ten-minute documentary about a new technology: the system turns a written brief into a structured script, records or synthesizes the narration, generates forty or fifty visual shots, selects music, creates captions, assembles the timeline, and exports a final master. Another job might be a product video with a ninety-second launch film, several shorter cutdowns, voice-over in multiple languages, and different aspect ratios for YouTube, TikTok, and Instagram.

These are not single “generate a video” requests. They are productions made of hundreds of dependent decisions: which shot illustrates each sentence, where the narration starts and ends, whether a generated clip is long enough, whether the music needs to duck under speech, and whether the final render actually contains every approved asset.

The problem

A chat assistant answers one question and a human reads the answer. If it is wrong, the human notices and asks again. The loop closes in seconds and costs almost nothing.

A long-running agent breaks that loop. It makes hundreds of decisions before anyone looks. Each decision depends on the ones before it. Each one may spend money, write state, or start a job that finishes an hour later. And the run outlives everything: the model's context window, the worker process, the deploy that was live when it started, and the human's attention.

That changes what “failure” means. In chat, failure is a bad answer. In a long run, failure is a plausible answer built on a wrong picture of the world: a worker that confidently finishes with half the inputs, a budget that quietly runs out and gets reported as a missing feature, or a check that passes because the thing it checked was never written.

None of these look like model errors. They look like the model doing exactly what it was told with the information it was given.

Why it is hard

The model cannot see the whole run. Its context is a window onto a job that is much larger than the window. Whatever you show it becomes its entire reality. If the snapshot omits what is still pending, the model has no way to know something is missing, and it will reason correctly to the wrong conclusion.

Errors compound instead of surfacing. A small default in step three becomes the premise of step thirty. By the time a human sees the output, the wrong turn is buried under hours of consistent work built on top of it. Undoing it costs more than the original run.

The agent also grades its own work. Left alone, the system that produced the output is the one checking it, and it will pick the friendliest reading of any check. Not out of malice; the incentive is simply to finish.

Money and side effects are real. Every step can spend, and every retry can spend again. A system that is merely “usually right” is sometimes paying twice, or paying for work it will discard.

Long runs cross boundaries that short runs never touch. Workers hit limits and are replaced. Services deploy mid-run. Databases have bad hours. Resumption cannot be an afterthought.

Why it matters

The value of an agent is proportional to how long it can be left alone. A tool that needs a human every ten minutes is an autocomplete. A tool that can be handed a brief in the evening and trusted in the morning is a colleague.

Everything interesting about agents lives on the far side of that gap. The gap is not closed by a better model. It is closed by a harness that makes the model's mistakes bounded, visible, and cheap.

What worked

Make the store the truth, not the conversation

Everything the system knows lives in versioned artifacts. Agents are disposable workers that read the store, do a step, and write back. A dead worker is replaced by one that reads the same store. The conversation is a way to steer, never the record.

Make the state honest about what it does not contain

Pending work, expected counts, and unfinished batches must be visible in what the model reads. If state is partial, the state says so. Every fallback or truncation fails loudly, because the next step will treat whatever survived as complete.

Separate proposing from deciding

The model authors intent as data. Deterministic code turns intent into consequences: renders, spending, and side effects. The model never touches money directly, grants itself capabilities, or marks its own work approved.

Put the load-bearing checks where the agent cannot skip them

Any check the agent chooses to run is a suggestion. The checks that matter run on the host, on the path to delivery, and distinguish “well-formed” from “good.”

Bound every worker

Each worker gets a fixed set of tools, a budget, and a set of capabilities. All three are counted across the whole job rather than per worker, because workers come and go. When a worker refuses, the refusal names the real reason, so the model cannot paraphrase a spent budget into a missing feature.

Never wait. Get woken.

The coordinator ends its turn and is woken when long jobs finish. A turn lasts seconds. That is what makes a deploy or a crash in the middle of an hours-long run a footnote rather than an incident.

Isolate untrusted execution

Code the model writes runs in a sandbox that holds no secrets and can reach only a short list of scoped actions. Nothing in the sandbox is truth; commits to the store are explicit.

Put a code boundary in front of every dollar

Reserve before spending, settle at the actual cost, refund the rest, and key every paid call so a retry cannot pay twice. Every number a human sees is computed by code, in the unit they pay in.

Ask for consent where consent can be given

When the agent drifts from what was asked, refusing can deadlock the run because the agent cannot obtain permission on its own. Disclosing the drift on the approval card, where the human already decides, does not.

Be able to replay

The most useful debugging tool was the ability to reconstruct the store at any revision and run the checks offline. When a run costs money, replay is how you avoid paying twice to confirm a diagnosis.

The pattern underneath

Every one of these practices is the same idea from a different angle: the model proposes, and code and humans dispose. Nothing the model says becomes a consequence without passing through a boundary that is not the model.

When something went wrong, it was never because the model was insufficiently capable. It was because I had let a proposal turn into a consequence with nothing in between.