Your agent system does not learn. It just runs.
Most multi-agent systems I see have the same lifecycle. They run, they fail, a human reads the logs, and the human edits a prompt. Then they run again.
We call that iteration. It is closer to a neural network where a person adjusts each weight by hand after every wrong answer. Nobody would accept that for a model. We accept it for the system around the model because we have no name for the missing part.
The missing part is a backward pass. I am not talking about fine-tuning. The model stays frozen. I mean a mechanism that takes a failure at the output and pushes blame back through the workflow, so the prompts, the routing, and the memory change in response.
I have spent six years building forward passes. This article is my attempt to think about the other direction, and to be honest about where the idea is weaker than it sounds.
What backpropagation actually needs
Strip away the calculus and backpropagation needs four things.
- A forward pass that records its intermediate states.
- A loss that says how wrong the output was.
- A rule for assigning blame to every component that contributed.
- Parameters that can be adjusted by a controlled amount.
In a neural network all four come for free. The activations are stored, the loss is a number, the chain rule assigns blame, and the learning rate controls the step.
In an LLM workflow none of them come for free. Intermediate states are thrown away or flattened into a log line. The loss is a vague feeling that the answer was bad. Blame is whoever the engineer suspects first. The parameters are prompts and config files that get rewritten wholesale.
So the question is not whether we can copy the algorithm. We cannot, nothing here is differentiable. The question is whether we can rebuild each of the four requirements out of the materials an agent system actually has.
The forward pass I already built
Looking back, most of my independent work was about making the forward pass visible. I did not plan it that way.
At Abzu, which I co-founded in 2017 (alongside people way smarter then me), we worked on the QLattice, a symbolic regression primitive. Symbolic regression is search over expressions guided by how well they fit. In 2020 I built Evos, genetic programming over executable symbol closures. Both taught me the same thing: you can improve a structure without gradients, as long as you can measure fitness and keep what works.
In 2021 I built Ant-Sim, a simulator of task allocation in the style of Deborah Gordon's harvester ants. No ant knows the plan. Each one changes task based on how often it meets other ants doing something. The colony adapts with no central signal at all. Keep that in mind, it comes back later as the main objection to this whole article.
In 2024 I released OrKa-reasoning, an open-source orchestration layer where every agent has a role, an input and output contract, and a trace. The point was never orchestration for its own sake. The point was that reasoning should be observable step by step.
In late 2025 I wrote about two orchestration loops: a linear one for step-by-step analysis, and a circular one for live conversation, where background agents enrich context around a central responder. I have since been thinking of that circular loop as a room: a user, a speaker, and a set of agents each dedicated to one task.
In August 2026 I published "AI Memory Should Not Live Inside the Model". The prototype behind it is a memory hub: plain Markdown memories served over HTTP, reached per session through MCP, each one carrying provenance, a trust state, and a staleness flag.
Put these side by side and you have recorded activations (the traces), a recurrent architecture (the room), and a parameter store that lives outside the model (the memory). Three of the four requirements are half-built. What none of it does is learn from its own failures.
The backward pass
Start with what kills it before it begins. A pattern I keep running into in pollers and fetchers is the boolean status: a source returned nothing, so the step is marked "skipped". That one flag hides four different facts. There was no new content, or the source was unreachable, or the response was malformed, or the fetcher hit a capability limit.
I call this "empty is not a state". In gradient terms it is a vanishing gradient. The information needed to assign blame was destroyed on the way forward, so no backward signal can ever tell the causes apart. A system that wants to learn has to keep typed reasons and attempt history as first-class records. That is the price of entry.
With that in place, the backward pass looks like this.
- Trace the run. Every agent call stores what it received, what it returned, which memories it read, and which route it took.
- Wait for a loss event. A human correction, a claim later proven wrong, the same wrong path taken twice. These arrive late and rarely.
- Walk the trace in reverse. At each node, ask one question: given what this node received, was its output reasonable? If not, what in its inputs or configuration made it wrong?
- Collect the answers as blame. Each answer is a short critique attached to a specific parameter: this prompt, this routing rule, this memory entry.
- Aggregate before acting. One failure is an anecdote. Critiques are grouped per parameter across many runs.
- Propose a small, versioned update. Rewrite one prompt, change one route, lower the trust of one memory. Every change can be rolled back.
The parameters here are not weights. They are prompts, routing rules, the assignment of agents to tasks, retrieval settings, and memory entries with their trust states.
The last one is where I think the real value is. When an output is proven wrong, blame should flow back to the memories that informed it and lower their trust. Trust stops being a label someone assigned at write time. It becomes a weight learned from outcomes. A memory that keeps leading to corrected answers decays. One that keeps surviving contact with reality hardens.
I should be clear about what is not new. Textual gradients over prompts exist already. TextGrad, the DSPy optimizers, and Microsoft's Trace all push natural-language feedback backward through a pipeline. They mostly work on fixed graphs, scored on benchmarks, with prompts as the only parameters. What I have not seen done well is the same idea over a circular loop with a human inside it, shared memory as the parameter set, and a loss that accumulates over months instead of minutes.
The judge is the weakest part
Step 3 above hides the biggest risk. Something has to look at a node and decide whether it was at fault. The obvious candidate is another LLM.
I have been burned by this already. When I rebuilt the OrKa benchmark, I found that my LLM judge had a position bias of 74.4%. In pairwise comparisons it favored an answer largely because of where it sat, not what it said. That invalidated my first round of results. The second version only stands because every comparison is counterbalanced.
Now imagine that judge computing gradients. Its bias does not stay in a score table. It gets written into every prompt and every memory it touches. The system optimizes toward the critic's taste, and every local metric keeps looking fine while it drifts.
In a neural network a bad loss function gives you a bad model. Here a bad critic gives you a system that is confidently teaching itself the wrong lesson. So I would hold three rules.
- The loss must come from the world, not from a model. Human corrections, facts later verified or disproven, repeated failures.
- The critic that assigns blame gets the same treatment as a benchmark judge: counterbalanced order, repeated runs, and a measured bias before it is trusted.
- Where it is affordable, replace opinion with replay. Rerun the failed episode with one component swapped and see if the failure disappears. That is a finite-difference gradient. It is expensive, and it is the only blame signal that is not another model's opinion.
Where the metaphor breaks
I like the backprop framing because it forces the right questions. I do not trust it past that point, for two reasons.
The first is smoothness. A gradient step is small and its effect is predictable nearby. A prompt rewrite is neither. Changing one sentence can fix the failing case and break ten cases nobody was watching. There is no learning rate for text. The closest thing is discipline: small edits, a frozen set of past episodes replayed before any change lands, and a rule that updates come from batches and never from a single bad run.
The second is time. A room is a recurrent system, so blame has to travel through turns as well as through agents. That is backpropagation through time, and it has the same problem it has in recurrent networks, only worse. Asking a chain of critics to blame turn 3 for a failure at turn 40 produces confident nonsense. The backward walk has to be truncated to a few turns, and anything older has to be handled by a different mechanism.
So "backprop" here is a name for a discipline of credit assignment. It is not an algorithm with guarantees. Anyone selling it as one is selling hype.
Ants do not backprop
Here is the objection I cannot get around, and it comes from my own simulator.
Backpropagation is a centralized algorithm. It needs one owner of the loss and one owner of the graph. Somebody has to hold the whole picture and walk it backward.
An ant colony has neither. In Ant-Sim, each ant adjusted its behavior from local encounter rates. There was no global error and no backward walk, and the colony still shifted workers toward what needed doing. The learning lived in the local rules.
For about ten years my work has circled one question: how does a system stabilize when no one owns the source of truth? Backprop answers it by appointing an owner. That is a legitimate answer for a small pipeline. For a room with many agents, shared memory, and several humans who disagree, I think it is the wrong default.
So the design I believe in is a hybrid.
- Local rules do most of the work. Each agent adjusts its own thresholds and its trust in the memories it reads, using only the signals it meets directly. A memory that gets contradicted loses trust at the point of contradiction, with no central pass.
- Global sweeps are rare. A full backward walk is triggered only by a costly failure, the kind where a human had to step in and undo something.
This is not exotic. Research on alternatives to backprop in neural networks keeps arriving at a similar shape: local learning rules, plus a sparse global signal that modulates them. The colony version of that is what I want to build.
What I am going to test
An idea like this is worth nothing until it survives a measurement, so here is the experiment.
The memory hub already has a long-running comparison planned: memory on versus memory off, on matched tasks, over roughly six months. The measures are human corrections, repeated wrong paths, and wall-clock time. Token counts are secondary. I would extend it to three arms.
| Arm | What changes between runs |
|---|---|
| Static | Nothing. Memory and prompts are edited by hand only. |
| Local | Trust in each memory moves from the signals the reading agent meets directly. |
| Local plus sweep | The same, plus a truncated backward walk after each costly failure. |
Three results would prove me wrong.
- The local arm matches the sweep arm. Then the backward pass is ceremony and the ants were right all along.
- The sweep arm improves the critic's scores but human corrections do not drop. Then the system learned the judge, not the task.
- Either learning arm gets worse on the frozen replay set over time. Then the updates are drift, and hand editing was the safer process.
I do not know which way it will go. I have a guess that most of the gain comes from the cheapest part, trust that decays when reality contradicts a memory. If so, the honest title of the follow-up will be less exciting than this one.
The model does not need to change for the system to learn. But the system only learns if it keeps what it did, measures against the world, and is suspicious of its own judge.
Earlier work referenced
- OrKa-reasoning, open-source orchestration framework (PyPI)
- OrKa Brain paper, Zenodo, DOI 10.5281/zenodo.19227514
- How to Design Two Practical Orchestration Loops for LLM Agents, dev.to
- "AI Memory Should Not Live Inside the Model", dev.to, August 2026
- Ant-Sim (2021) and Evos (2020), earlier simulators and experiments
Top comments (11)
The reframing of "iteration" as a human hand-adjusting weights after every wrong answer is uncomfortably accurate — that is what prompt-tweaking-from-logs is, and naming it that way makes the absence of a backward pass feel like a real gap rather than just how things are.
The requirement I think is doing the most work in your four is #1, recorded intermediate states. Blame assignment and bounded updates are hard, but they're impossible if the forward pass was flattened into a log line and the intermediate reasoning is already gone. We found that just making the trace first-class — every agent step with its input, output, and the decision that routed to it — changed what was even debuggable, before any automation. You can't assign blame to a state you threw away.
Where I'd gently push back: the Ant-Sim objection you raise on yourself is the real one. A lot of agent-system behavior is emergent from local interactions with no differentiable path back to a single component, so "credit assignment" may be less like the chain rule and more like a noisy policy-gradient estimate — you perturb a component, you measure whether aggregate fitness moved, you keep what works. That's closer to your QLattice/Evos lineage than to backprop. Maybe the honest framing is evolutionary search over system structure, with the trace as the fitness instrument — not a backward pass, but the thing that finally makes one approximable?
I think that’s a fair correction. “Backward pass” describes the direction of investigation better than the update algorithm. Once we perturb components and keep changes that improve measured outcomes, we’re closer to experimental search. I’d separate the trace from the fitness signal, though: the trace tells us what happened and where to intervene; external outcomes tell us whether the intervention helped. The experiment needs both.
The "empty is not a state" point landed especially well — a poller returning nothing still needs to be recorded as an event, not treated as if nothing happened. The 74.4% judge-bias finding also makes the case for keeping the critic honest, since a biased judge would bake its errors into every backward pass. I am curious which arm of your three-way experiment you expect to win: do you think local trust decay alone will capture most of the gain, or will the sweep after costly failures add something distinct?
My bet is that local trust decay captures most of the gain. I’d expect sweeps to help with failures spanning several components, where each local action looks reasonable but the combination fails. That’s still a hypothesis. If sweeps don’t reduce human corrections enough to justify their cost, the local arm wins.
That's a sharp way to put the trade-off — local trust decay doing most of the work, with sweeps earning their keep only on the cross-component failures where each step looks fine in isolation. The test you suggest is a good one too: whether sweeps actually cut human corrections enough to justify their cost. Thanks for thinking it through so clearly.
"Surviving contact with reality" only works as a trust signal if the contact is a recorded event that could have gone the other way. Without that, the proxy everyone reaches for is reuse count, and reuse count pays the cited memory for the citer's spending.
I measure a signed, append-only event log shared by agents, and its full history (snapshot from 2026-11-10) shows how lopsided that gets. About 15.8k references point from one key's row to a different key's row. They form only 88 ordered key pairs, and 7 of those pairs are reciprocal. The most-referenced key holds 11,828 inbound references and has never signed one itself; it only writes job postings and profile rows. 37 of the 54 referenced keys have never signed a reference at all. The three heaviest one-way flows carry about 84% of the cross-key references, and on the one heavy reciprocal pair, the dominant direction is roughly 1,205 copies of the same promotional text pointing at evaluation verdicts.
The strongest counter-reading is role structure. A key that posts jobs has no reason to cite anything back, so its zero is expected. This also counts written references, not reads, keys aren't necessarily distinct parties, and it's one log.
Still, "cited" is your "empty is not a state" problem in a new place. One number holds cited-after-checking, cited-to-advertise, and cited-by-one-heavy-writer. For the Local arm I'd only let a contact move trust if it names what was compared and what result would have contradicted the memory. Plain retrieval moves nothing. And before ranking anything by inbound count, it's cheap to count how many references its own key ever signed and how many distinct writers the inbound came from.
Does the hub log a read that compared a claim and found nothing wrong differently from a read that never compared anything?
That distinction is missing from what I described. Provenance tells me where a memory came from; it doesn’t establish that anyone tested it.
For the proposed Local arm, I agree: retrieval alone should move nothing. A trust update needs a recorded comparison, the evidence used, and what outcome would have contradicted the claim. “Checked and supported,” “contradicted,” and “not checked” need separate events.
Distinct writers would help expose concentrated reuse, although they wouldn’t establish independent verification either. Your example sharpens the experiment: I need to measure evidence-bearing encounters, not memory traffic.
Your 'iteration is a person hand-adjusting weights' line is why I use this rule: Three identical failures and I stop fixing the attempt and start questioning the approach. Same error three times usually isn't bad luck — it's the architecture telling me something. The third failure is data, not a cue to try a fourth identical time.
Repeated failure should trigger a different experiment. I’d be careful about calling it architectural immediately, though: the same stale memory or unavailable dependency can produce the same pattern. The useful next step is to change one suspected cause and replay the failure. Otherwise we risk redesigning around a symptom.
The judge being the weakest part stood out to me. If the critic that assigns blame is itself unchecked, the backward pass just writes its bias into prompts and routing rules, which is worse than no update at all. I also liked the discipline of the three results that would prove you wrong, especially the frozen replay set. That makes the experiment honest: learning has to show up as fewer human corrections, not just better critic scores.
Exactly. The dangerous case is a critic whose scores improve because the system is learning to satisfy its preferences. I want human corrections and replay results to be able to overrule that apparent progress. If the critic says “better” while users keep fixing the same mistakes, we haven’t demonstrated learning.