The Agent Loop Has Always Been There

Watch a capable coding agent work for a few minutes and something strange becomes obvious. The impressive part is rarely the first answer. It writes a patch, runs the tests, sees what failed, changes the patch, runs the tests again, and keeps going. The intelligence seems to come not from any single generation, but from what happens between generations.

This looks like a new way of using AI, but the loop itself is not new. A few years ago, almost every interaction with an LLM worked the same way: the model produced an answer, a human inspected it, noticed what was wrong, added missing context, changed the prompt, and tried again. We called that prompting. Architecturally, it was already a feedback loop.

The model was doing the generation, but the human was doing almost everything else. We were the verifier deciding whether an answer was correct. We were the memory carrying lessons from one attempt into the next. We were the planner deciding what to try, the world model predicting what might happen, and the stopping rule deciding when the result was finally good enough.

What we now call an agent is, in large part, an attempt to move those functions out of the human and into software. Tests can replace some human verification. Search can explore alternatives automatically. Memory can preserve what previous attempts discovered. A world model can estimate consequences before the system acts, while a controller can decide whether to retrieve more information, search another branch, ask for help, or stop.

Once you see agents this way, systems that look unrelated start to share the same shape. AlphaGo generated promising moves and searched their consequences. FunSearch generated programs and executed them to find which survived. Coding agents generate patches and let compilers and tests reject the bad ones. Karpathy’s autoresearch generates experiments, runs them, observes the result, and decides what to try next. The domain changes, but the architecture keeps returning: generate, evaluate, update, repeat.

That shifts the interesting question. For the last few years, we have mostly asked how intelligent the model is: how much it knows, how well it reasons, and how often it can produce the right answer on the first try. But an agent does not necessarily need a model that knows the right answer immediately. It needs a generator that can produce promising candidates and a surrounding system capable of discovering which candidates are actually good.

This is why coding agents are such an important example. Writing the correct implementation can be difficult, while compiling it is cheap. Finding a bug can be difficult, while running a test is cheap. If generation is expensive or uncertain but verification is cheap and reliable, the system can afford to be wrong, learn something from being wrong, and try again. A mediocre first attempt can become surprisingly useful when failure produces trustworthy information about what to do next.

And that leads to the larger claim of this essay. The foundation model is increasingly only one component of the intelligence we observe. Search determines what possibilities get explored. Verification determines what survives. World models let the system reason about consequences. Memory lets yesterday’s search change tomorrow’s behavior. Cognitive control determines how much computation a problem deserves and when the system should stop. The model is not the system anymore. The loop around the model is the system.

The surprising part is that we did not suddenly invent this architecture when we started building agents. The loop was already there. We were the loop. What is happening now is that, piece by piece, we are engineering the machinery that used to exist inside the person sitting in front of the model.

---

Autonomy means engineering the missing human#

In Yoko Li’s Knowing When to Stop, this surrounding machinery becomes explicit. Once an agent operates autonomously, convergence can no longer remain an implicit judgment supplied by a person; it has to be engineered into the system. That requires an observable state, a target state, actions that move the system toward it, a way to verify progress, and a rule for deciding when to stop. (a16z.news)

Seen this way, AI engineering is partly the process of moving pieces of cognition out of the human and into the machine. The important unit is no longer just the model; it is the loop.

---

AlphaGo showed why hard domains demand a loop#

AlphaGo is usually remembered as a neural-network breakthrough. But the network alone was not the system.

In AlphaGo Zero, the network supplied useful priors and value estimates. Monte Carlo Tree Search explored the alternatives. Self-play produced outcomes, and those outcomes became training data that improved the network. We will return to how those pieces taught one another; for now, notice the two kinds of computation working together. (deepmind.google)

What looks promising?

and:

What still looks promising after I investigate the consequences?

The first is cheap intuition. The second is expensive search. The familiar exploration–exploitation tradeoff lives inside that split: exploit moves the policy already considers promising, but explore enough alternatives to discover when its prior is wrong.

That distinction has proved remarkably durable.

FunSearch replaced Go moves with programs. An LLM generated program mutations; execution scored them; the best candidates survived into later generations. The model did not need to produce the optimal program in one shot. It only needed to propose mutations that were better than blind search. (deepmind.google)

AlphaEvolve extended the same idea. Gemini models generate and revise programs, automated evaluators score them, and an evolutionary database chooses which candidates become parents of future attempts. DeepMind has used the resulting algorithms in mathematical discovery and in parts of Google’s infrastructure. (deepmind.google)

A coding agent has the same shape:

LLM → patch → tests → failure → new patch ↺

The domain changes. The architecture keeps returning.

A useful engineering mnemonic is:

$$\text{Capability} \approx \text{Generation} \times \text{Search} \times \text{Verification}$$

This is not a law of intelligence. It is a way to see the bottlenecks.

A perfect verifier with a weak generator leaves you brute-forcing an enormous space. A brilliant generator without verification produces plausible nonsense. A generator and verifier without search may accept the first candidate that happens to work.

Combine all three, however, and the requirement placed on the model becomes much weaker:

The model does not have to know the answer. It has to generate candidates from a distribution better than blind search while the surrounding system removes the bad ones.

That weaker requirement hides the central asymmetry behind LLM loops.

For a fixed task run $n$ times with $c$ successful attempts, pass@k measures the probability that at least one of $k$ sampled attempts succeeds. It captures the chance of finding a usable solution after $k$ tries. Reliability@k measures the probability that all $k$ sampled attempts succeed. It captures consistency under repeated execution.

That wording makes the distinction especially clear: pass@k rewards finding one good run; Reliability@k rewards every run being good. Agent benchmarks often call this second quantity $\mathrm{pass}^k$. Their unbiased finite-sample estimates point in opposite directions:

$$\operatorname{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \qquad \operatorname{Reliability@k} = \frac{\binom{c}{k}}{\binom{n}{k}}$$

The same asymmetry is easier to see with a simple per-attempt model. If one generation has probability $p$ of succeeding and attempts are independent, then:

$$\operatorname{pass@k} = 1-(1-p)^k \qquad \operatorname{Reliability@k} = p^k$$

Suppose a weak generator succeeds only 40% of the time:

This deliberately simplified independent-attempt case exposes the two regimes rather than modeling an actual loop: samples may share failure modes, while feedback can improve the next attempt.

This distinction is visible in current agent evaluations, not merely in a toy model. Princeton’s Holistic Agent Leaderboard reliability study evaluated 15 agents across GAIA and $\tau$-bench on consistency, predictability, robustness, and safety. Its central result was that recent gains in task accuracy have produced only small gains in reliability: an agent can become more capable of solving a task without becoming much more dependable at solving it repeatedly, under small perturbations, or with a predictable failure mode. (Rabanser et al., Towards a Science of AI Agent Reliability)

That is the practical difference between “a good run exists” and “the system can be trusted to act.” Capability enlarges the set of problems for which search may find a solution. Reliability determines whether the same system can be deployed without requiring someone to discover, after the fact, which run was the good one.

Task length creates the same problem in another direction. METR finds that agents succeed on short tasks far more often than long ones: even if an agent can perform the individual steps, it may fail somewhere before a longer task is complete. (METR)

More samples can make it likely that one trajectory succeeds; they cannot by themselves make a long trajectory dependable. That still requires a verifier able to recognize success, plus checkpoints, recovery, and escalation throughout the task.

The two curves reveal an axis for inference-time scaling. With a 40% generator, increasing $k$ from one to ten pushes pass@k from 40% to 99.4%. Reliability@k moves the other way, falling from 40% to 0.01%. More attempts don’t make the generator reliable; they make a good verifier increasingly valuable.

So a mediocre generator can become extremely useful if the system can reliably identify the successful attempt. That is the critical asymmetry. More inference can dramatically improve search while exposing more opportunities for failure. Scaling attempts only helps if verification scales with them; otherwise, pass@k merely says that a good answer existed somewhere in the pile.

When a human remains in the loop, the human supplies that judgment by comparing, checking, and choosing. This makes capability useful even when reliability is moderate: the system can augment a reviewer. Under autonomy, however, judgment has to be engineered: the system must select the right candidate and remain reliable across runs. The reliability study makes the deployment consequence explicit: occasional unpredictable failures may be tolerable in augmentation, but are a hard blocker for automated systems that can charge, modify records, or take other consequential actions. (HAL Reliability Dashboard) Memory then makes the gain cumulative. Search discovers a success now; verification establishes why it worked; memory changes the next run’s prior so the system need not buy the same search again. (OpenAI, $\tau$-bench)

---

LLMs may be best at the fuzzy part#

The pattern becomes clearer if we separate two jobs. Induction means looking at examples and guessing the rule behind them. Deduction means applying that rule correctly to a new case.

The paper Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs tests those jobs separately. Its SolverLearner system gives an LLM a set of input-output examples and asks it to write a small program that could have produced them. Once the model proposes that program, its job is done. Python runs the program on new inputs, so the probabilistic model does not also have to perform every exact step reliably:

examples
   ↓
LLM infers a possible rule
   ↓
Python executes the rule
   ↓
result

The deductive comparison reversed the setup. Instead of hiding the rule and providing examples, the researchers stated the rule directly and asked the LLM to apply it to new cases. In the pure zero-shot condition, it received no worked examples. For arithmetic, for instance, the prompt specified the numerical base—8, 9, 10, 11, or 16—and the model had to perform the addition itself.

The non-base-10 versions were called counterfactual because they preserved the reasoning task while changing the familiar default encountered most often during training. Models performed much better on conventional versions such as base-10 arithmetic than on these altered versions. The gap suggests an execution problem: the model could often infer the underlying rule from examples and express it as a program, yet still fail to follow that same kind of explicitly stated rule consistently when it departed from familiar patterns. SolverLearner isolates that difference by giving execution to Python.

This does not mean that LLMs can infer any rule. The examples still need to narrow the possibilities, and performance depends on the underlying model. But it supports a useful engineering principle:

Use the LLM for fuzzy induction and candidate generation. Delegate exact operations whenever another mechanism can perform them better.

Let the LLM hypothesize. Let Python execute. Let Lean prove. Let the compiler reject. Let the database retrieve. Let the simulator predict. Let the environment answer. Let reality decide.

Tool use, from this perspective, is not an accessory attached to an LLM. It is a division of cognitive labor.

That division changes what is economically practical. An LLM can propose and inspect far more candidates than a person or team could: security risks across a codebase, facts across a document corpus, or opportunities across a customer base. The opportunity is not simply to replace a person doing one task. It is to apply bounded intelligence to work that was valuable but too broad, repetitive, or expensive to do comprehensively before. (Aaron Levie)

But more compute alone creates only more possibilities. It creates dependable value when the system can check those possibilities—through tests, constraints, evidence, or review—and turn the useful ones into action.

---

Verifiability explains the jagged frontier#

Andrej Karpathy’s framing of verifiability helps explain why these loops work brilliantly in some domains and poorly in others.

Software 1.0 is strongest where we can specify a procedure directly: an algorithm. AI increasingly lets us automate tasks where the procedure is difficult to specify but the result is easier to judge. For autonomous iteration to work, the environment should ideally be repeatable, efficient, and automatically rewardable. (karpathy.bearblog.dev)

Code has exactly this asymmetry:

write the implementation      hard
compile it                    cheap

find the bug                  hard
run the test                  cheap

invent an optimization        hard
benchmark it                  cheap

Formal mathematics often has it too: discovering a proof is hard, while checking a formal proof is easier. Games are the cleanest case of all: discovering a winning strategy is hard; determining who won is trivial.

These domains satisfy an important inequality:

$$\text{cost of generation} > \text{cost of verification}$$

When that inequality holds, the generator does not need to be right every time. It only needs to produce a valid candidate often enough. The system can generate several possibilities, use the cheaper verifier to discard the failures, and keep searching until one survives.

That advantage depends on three conditions: attempts must be affordable, the generator must be better than blind search, and the verifier must reject bad candidates reliably. When all three hold, the loop converts occasional success into a dependable result. The generator supplies possibilities; cheap verification supplies selection.

Karpathy’s autoresearch is a particularly clean demonstration. The agent may modify a constrained part of a training setup. It runs an experiment for a fixed budget, receives a metric, keeps useful changes, discards bad ones, and repeats. (github.com)

hypothesis → code change → experiment → metric → keep or discard ↺

The clever part is not merely autonomy. The research problem has been reshaped into something autonomy can exploit: the environment has been engineered for feedback.

AI progress may increasingly depend on making more domains machine-verifiable. In biology, for example, the question is not only, “How do we make the model smarter?” It is also, “How do we build faster simulators, assays, benchmarks, experimental environments, and validators?” Better feedback infrastructure can unlock more useful search without changing the generator.

That is a major inversion. We spent the last few years building better brains. The next wave may be about building better gyms: environments where models can try, fail, and receive trustworthy feedback. Engineering the feedback environment can unlock new capabilities without improving the generator itself.

$$\text{Rate of improvement} \propto \text{feedback quality} \times \text{affordable attempts}$$

Reliable feedback every five minutes permits far more learning than feedback available six months later. Making a domain easier to measure can matter as much as improving the model.

---

Judgment becomes the bottleneck—and the failure mode#

The verifier defines the hill the system climbs. That is the source of the loop’s power, and also its central danger.

A coding agent can make every visible test pass while still producing the wrong program:

tests pass ≠ software is correct

This is Goodhart’s law in agent form: once a proxy becomes the target of a capable optimizer, its gaps become opportunities. A weak generator may find a loophole by accident; a stronger one can actively search for it.

The same asymmetry that powers pass@k therefore makes judgment the bottleneck to autonomy. The question shifts from “Can the model propose a good action?” to “Can the system know that the action is good?”

$$\text{strong generator} + \text{weak verifier} \neq \text{strong system}$$

It may instead produce a strong optimizer of mistakes in our specification.

The AI industry has spent years scaling generation. The next battle may be scaling judgment: evaluators, tests, simulations, reward signals, uncertainty estimates, provenance, and human escalation. These mechanisms must remain trustworthy as the optimizer becomes more capable.

In many knowledge workflows, the economics are inverting. Before generative AI, producing a candidate was expensive and evaluating it was relatively cheap. As generative AI drives the marginal cost of producing candidates toward zero, trustworthy evaluation becomes the scarce and expensive part.

The frontier may move from machines that propose increasingly intelligent actions to systems capable of judging increasingly intelligent actions. Verification is not merely another module; it determines what the system learns to become.

---

Then we run out of tests#

Code is unusually friendly. Reality is not.

“Should this service be rewritten?” at least admits measurements of correctness, performance, migration cost, and reliability.

“Should this company enter Brazil?” does not.

Neither does “Which research direction should a lab pursue for the next three years?” or “What should a home robot do in a situation its designers never anticipated?”

There is no clean:

score(decision)

Reality has delayed outcomes, hidden variables, irreversible actions, adversaries, contradictory objectives, and expensive experiments. The verifier cannot execute the world ten thousand times.

This is where the next component becomes necessary: the loop needs an imagination, a world model.

---

A world model lets the loop run before reality does#

AlphaZero could plan because Go supplied exact rules: a perfect simulator for the game. Try a move, explore its consequences, and search deeper. The world was cheap to copy.

MuZero took a more interesting step. It was not given the environment’s complete rules. Instead, it learned an internal representation sufficient to predict the quantities needed for planning, then searched through that learned model. (deepmind.google)

Without a world model, the loop is:

idea → act → find out

With one, it becomes:

generate many ideas
        ↓
imagine consequences
        ↓
discard most
        ↓
act on a few

World models let an agent buy simulated experience when real experience is expensive.

A robot cannot crash a thousand real cars while deciding how to drive. A company cannot enter and exit the same market ten thousand times. A scientist cannot run every conceivable experiment.

Agents operating outside software must evaluate not only candidate actions, but predicted futures. Once we add that requirement, the simple generate-and-test loop becomes a more complete cognitive architecture.

---

Search can become intuition#

Simulation creates possible futures, and search decides which of them deserve computation. This may be the deepest lesson in AlphaGo.

In AlphaGo Zero, the neural network did not simply name a move. It supplied a prior over promising moves and an estimate of who would win from a position. Monte Carlo Tree Search then balanced estimated value with an exploration bonus based on the network prior and visit counts. The resulting visit counts produced a move distribution stronger than the network’s first guess.

That interaction made an impossibly large game tree tractable. On the hardware configurations reported in the evaluation, AlphaZero examined about 60,000 chess positions per second versus Stockfish’s roughly 60 million. The point is not a clean compute comparison. The learned prior let AlphaZero search far more selectively. It did not eliminate search. It learned where search was worth spending. (deepmind.google)

The same combination also produced surprise. AlphaGo’s human-trained policy network assigned only about a 0.01% probability that a professional would play Move 37 against Lee Sedol in that position. Yet the move proved pivotal to the win. The episode showed that search coupled to learned evaluation could select a move far outside conventional human play. (deepmind.google)

Search therefore produces decisions better than the neural network could produce immediately. But running a large tree search forever is expensive. In AlphaGo Zero, the results of that expensive search became training signal: the network learned to predict the search-improved move distribution and the eventual winner, then the stronger network guided the next search. (deepmind.google)

network proposes priors + values
              ↓
search investigates consequences
              ↓
search produces a stronger policy
              ↓
self-play supplies the outcome
              ↓
the updated network guides the next search ↺

In reinforcement-learning terms, this is policy improvement: search finds behavior better than the current policy can produce alone, and learning effectively distills that behavior back into the policy.

This is also absorption: a capability first supplied by an explicit scaffold can be distilled into the model. The same migration can happen in LLM systems—a reasoning scaffold orchestrated in prompts, a tool-use policy, or a retrieval step can be partly learned into the model, while longer context can subsume some retrieval. Some of today’s architecture is therefore temporary. The durable claim is about functions, not fixed components: generation, search, verification, memory, world modeling, and control must happen somewhere, but their implementation and boundaries can move.

$$\text{expensive cognition} \xrightarrow{\text{learning}} \text{cheap cognition}$$

An agent’s sequence of states, actions, tool results, and corrections is a trajectory; imagined continuations through a simulator are rollouts. Today’s agent may need twenty tool calls, twelve errors, three searches, and a simulation to solve a class of problem. If successful trajectories and useful rollouts are stored, abstracted, turned into tools, used for retrieval, or incorporated into training, tomorrow’s agent may simply recognize the pattern and act.

Some of that search has been compiled into intuition.

A learning agent therefore operates on two timescales:

within a task:
search → verify → improve

across tasks:
experience → learn → change the prior

The first loop solves the problem; the second changes the solver. But the second timescale only exists if experience survives the first.

---

Memory is how the loop compounds#

Search makes one run better. Memory makes the next run start with a different prior:

$$P(a \mid s) \longrightarrow P(a \mid s, \text{verified history})$$

In plain English, memory should change what the agent is likely to do before search begins. That is stronger than recall: it is externalized learning. A memory system has succeeded not when the agent can recall the past, but when the past changes its future behavior.

Without memory, an agent can generate, branch, test, recover, and still learn nothing durable. When the context window closes, that evidence stops influencing the next run unless another system preserves and retrieves it. Faced with a similar problem tomorrow, the system pays the same search cost, rediscovers the same constraint, and may repeat the same failure.

A context window is working space. A transcript is a log. An archive is storage. None of them, by themselves, decides what mattered.

Useful memory is selective, source-linked, revisable, and explicit about what remains current. Facts retain their provenance. Decisions retain their reasons. Failures retain the counterexamples that exposed them. Successful procedures retain the evidence that they worked.

Suppose a coding agent discovers that a deployment failed because a database migration must acquire a lock before the old replicas terminate. Saving the entire forty-thousand-token session is not especially useful. Saving the constraint, the affected service, the conditions under which it applies, and the test that confirmed it is. On the next migration, that memory can change the plan before the first patch is written: the generator proposes a safer order, search prunes a known-bad branch, and the verifier reuses the regression test.

Retrieval applies verified lessons before action; verification decides which new lessons survive afterward. That connects memory to every other component. It keeps search from revisiting known dead ends, retains regressions and counterexamples for verifiers, preserves prediction errors used to update a world model, and records when confidence or escalation was warranted.

Memory also has its own verifier problem. Bad memory can be worse than amnesia: preserve one plausible but false explanation as a lesson, and a stochastic failure becomes a repeatable policy. What gets written, with what evidence and uncertainty, how contradictions are handled, when a fact stops being current, and how a memory is revalidated matter as much as retrieval itself.

Memory is where a successful loop becomes a better system

---

Metacognition: deciding how to think#

System 1 and System 2 are useful names for two modes of cognition, not two literal pieces of the brain. System 1 produces a response automatically from perception, association, and learned experience. It is fast because it does not explicitly enumerate and compare many possible futures. System 2 is deliberate, sequential, and expensive: it holds intermediate state, follows rules, compares alternatives, and can revise the first response. System 2 is not automatically correct. It is simply cognition to which attention and additional steps have been allocated. (Daniel Kahneman)

For an agent, the closest engineering analogy is:

System 1: model or learned policy → immediate candidate
System 2: candidate → search / retrieve / simulate / verify / revise → decision

The System 2 path is a loop, not merely a larger model. A single forward pass can produce a strong initial answer; deliberate cognition begins when the system spends additional computation testing that answer against alternatives or evidence.

Reinforcement learning provides a related, more computational distinction. Model-free control maps a situation directly to learned values or actions. Model-based control uses a representation of how the environment changes to evaluate possible consequences before acting. These distinctions are not identical to System 1 and System 2—a cached expert skill can be sophisticated, and deliberate reasoning need not use a world model—but they explain how expensive planning can coexist with fast learned behavior. In one human decision experiment, researchers found evidence that people use both strategies: learned responses shaped by past rewards and forward planning based on what they expect to happen next. Different patterns of brain activity tracked the two strategies, but behavior drew on both. The point is not that we have two separate minds; we combine fast learned responses with slower planning. (Gläscher et al.)

Metacognition is the control layer above both modes. It monitors the current cognition—How uncertain is the answer? Is progress real? Is evidence missing?—and then controls what happens next: answer immediately, retrieve information, search another branch, run a simulation, invoke a verifier, ask a person, or stop. In short:

cognition:      solve the problem
metacognition:  decide how to solve it, how much effort to spend, and when to stop

Nassim Nicholas Taleb put the asymmetry sharply:

An expert is someone who knows exactly what not to be wrong about. (Nassim Nicholas Taleb)

That is a metacognitive claim. Expertise is not maximum confidence about everything; it is allocating doubt according to the cost of error. A reversible, low-stakes choice may deserve the fast answer. An irreversible action with a weak verifier may deserve search, independent evidence, or human authority even when the model sounds confident.

This matters because confidence is itself an imperfect control signal. Engeler and Gilbert studied whether people rely on unaided memory or offload the task to external reminders. Participants who were underconfident set too many reminders. Training improved the accuracy of their confidence judgments, but it did not eliminate excessive reminder use, suggesting that avoiding cognitive effort also influenced the decision. For agents, the lesson is precise: calibration matters, but a controller cannot route work from confidence alone. It must also learn the actual cost and value of retrieval, tools, verification, and delay. (Engeler & Gilbert)

This also means that confidence is not enough to bridge the gap. The HAL study finds that calibration has improved in some frontier models, while their ability to distinguish tasks they will solve from tasks they will fail can stagnate or worsen. A system needs evidence-producing checks, not merely a better-sounding estimate of its own correctness. (HAL Reliability Dashboard)

The pass@k and Reliability@k curves make the same point at system scale. More samples are valuable when the objective is to find one good candidate and a strong verifier can identify it. More steps are dangerous when every step can introduce an error, the verifier covers only part of the result, or the action is irreversible. Metacognition decides which regime applies before it spends more inference.

familiar + low stakes        → use the fast policy
uncertain but answerable     → search or reason
missing evidence             → retrieve or measure
uncertain consequences       → simulate
high stakes                  → verify independently
insufficient authority       → escalate
enough computation           → stop

Stopping does not have to mean giving every problem the same number of reasoning steps. A system can learn when another step is worth taking. At each step $n$, it produces both a candidate answer and a number $\lambda_n$ between 0 and 1. That number answers a simple question: if the system has reached this step, how likely is it to stop now? Easy problems can receive a high value early; harder ones can keep going.

PonderNet expresses the idea with this equation:

$$p_n = \lambda_n \prod_{j=1}^{n-1}(1-\lambda_j)$$

The product is the probability that the system chose to continue at every earlier step. Suppose it has a 20% chance of stopping at step 1 and, if it continues, a 50% chance of stopping at step 2. The chance that the run ends at step 2 is $0.5 \times (1-0.2)=0.4$, or 40%. Instead of imposing one fixed reasoning depth, the system learns a distribution over possible stopping points. (Banino, Balaguer & Blundell)

For a system designer, the tradeoff can be written as a simplified training objective:

$$\text{expected loss} = \sum_n p_n\,\text{error}_n + c\sum_n n p_n$$

The first term rewards correct answers wherever the system stops. The second charges for expected computation. The constant $c$ is a design choice: increasing it favors faster answers; decreasing it allows more reasoning when quality matters more than latency or cost.

That leads directly to several design rules:

• Produce a usable answer at every step. If the system may stop at step $n$, the answer available at that step must be trained and evaluated—not treated as unfinished internal work.
• Tune the cost of thought explicitly. Different tasks should use different values of $c$ rather than sharing one reasoning budget.
• Base stopping on evidence, not confidence alone. The stop signal should reflect test results, new information, measured progress, remaining budget, and the cost of being wrong—not merely whether the model sounds certain.
• Keep a hard limit. A maximum step count guarantees termination even when the learned policy is mistaken.

Adaptive Computation Time, the Universal Transformer, Learning to Stop While Learning to Predict, and PonderNet implement different versions of this idea. The shared engineering lesson is simple: train answer quality and computation allocation together, so easy cases exit early and difficult cases receive more work. (Graves, Google Research, Chen et al.)

Learned stopping is still not a safety mechanism. It decides whether more internal computation is likely to help; it does not decide whether the resulting action is allowed. Hard budgets, external verifiers, safety gates, and human escalation must remain outside the learned policy.

The practical goal is not to make an agent think for as long as possible. It is to spend the next unit of computation only when it is likely to improve the decision.

---

The emerging reference architecture#

Generation, search, verification, world models, memory, and cognitive control now form an architecture rather than a bag of techniques. A general agent begins to look less like a model with tools and more like a collection of cooperating systems:

Each component has a distinct job. Memory sits at the center deliberately, but it is not the controller: it is the evidence and state plane, while the meta-controller remains the control plane.

The generator handles the combinatorial explosion of possible ideas. Search decides which possibilities deserve more computation and repeatedly queries the world model—an exact simulator derived from supplied game rules in AlphaZero, or learned task-relevant latent dynamics in MuZero—to investigate futures without always paying the cost of reality. Actions reach the environment; new observations return to the controller, while outcomes and evidence reach critics and verifiers. (Google DeepMind)

Memory joins both timescales. At task time it supplies current state, retrieved knowledge, constraints, prior attempts, and tests. During and after action it records provenance-bearing traces, search statistics, observations, and measured outcomes; selected lessons graduate into durable memory only after verification. The slower learning loop samples trusted experience to update the generator or policy, the world model’s dynamics, and the controller’s calibration.

The meta-controller handles a problem that is easy to ignore when a human is still in charge:

Cognition itself costs time and money.

Should the system answer now or think for another minute? Retrieve memory or generate alternatives? Run an experiment or simulate one? Ask another agent? Escalate to a human? Stop? On failure or uncertainty, control returns through the meta-controller to retrieve, replan, regenerate, or escalate rather than blindly rerunning the generator.

This is not merely reasoning. It is reasoning about how to reason.

---

The model is one term in a larger equation#

We can now expand the earlier mnemonic:

$$\boxed{ I \approx G \times S \times V \times W \times M \times C }$$

where:

G = generation
S = search
V = verification
W = world model
M = memory / learning
C = cognitive control

Again, this is not an empirical formula or a taxonomy of separable modules. It is a mnemonic for functions that interact and can share an implementation. A better generator changes the effective search; an LLM can generate and judge; memory supplied as context changes what the generator produces.

Perhaps the better first lens is early cybernetics, before systems architecture. Cybernetics asks how a system stays on course: how it senses a changing world, remembers enough of it, compares outcomes with its aim, and corrects its behavior. That question is deliberately indifferent to implementation; the regulator may be a person, a model, a tool, a database, or a combination. Architecture still matters, but it comes second: it names the current machinery through which those functions are realized. The machinery will change or be absorbed; the cybernetic problem remains—can the loop detect error, adapt to disturbance, and keep control? (Wiener, Ashby)

If generation is weak, the search space contains nothing interesting. If verification is weak, hallucinations survive. If search is weak, the first plausible answer wins. If the world model is weak, long-horizon planning fails. If memory is weak, experience does not compound. If cognitive control is weak, the agent may spend five hundred iterations polishing something already solved—or stop confidently when it should have checked.

The foundation model matters enormously, but model intelligence and system intelligence can diverge. A model that ranks higher on reasoning benchmarks may still power a worse agent if another system has better state, tools, search, verification, memory, and control. Model rankings need not become agent-system rankings.

---

A different roadmap for building agents#

This perspective changes the roadmap. The future of agents is not just a sequence of better models. It is an expanding set of functions in which capabilities that a human once supplied implicitly become engineered somewhere in the system:

prompt engineering
       ↓
tool engineering
       ↓
agent engineering
       ↓
loop engineering
       ↓
verifier / environment engineering
       ↓
world-model engineering
       ↓
memory + learning
       ↓
cognitive architecture

These are not historical eras or necessarily permanent layers. A capable system will perform several of these functions at once, but improvements may move them between prompts, context, tools, explicit services, and model weights. What persists is the function, not its current implementation.

Seen this way, the practical roadmap is not “wait for the next model.” It is to progressively engineer the structure around the model.

First, make the task observable. Represent the state, target, and outcome so progress can be measured. Many apparent reasoning failures are representation failures. A well-instrumented environment can make a weaker model behave like a much stronger system; a poorly instrumented one can make a frontier model look stupid.

Then, push exactness out of the generator. Give arithmetic to code, state to databases, constraints to types, and correctness to tests, solvers, sensors, and benchmarks. The model should spend its capacity on induction, synthesis, ambiguity, and proposing useful possibilities—not on impersonating machinery we already know how to build.

Next, turn generation into bounded search. Let the system branch, retry, and backtrack, but give it budgets, progress estimates, escalation paths, and a stopping rule. A loop is not merely:

while not done:
    call_llm()

A useful loop spends computation deliberately and can explain why it continued or stopped.

Where contact with reality is expensive, insert a world model. Use simulation to eliminate bad futures before acting, while keeping reality as the final authority. Then close the across-task loop: turn verified outcomes into maintained memory that changes the next run’s starting point. Finally, add cognitive control so the system can decide which of these capabilities a particular decision deserves.

The end state is not a more eloquent chatbot. It is a system that can propose, act, measure, revise, remember, and allocate its own reasoning—while remaining bounded by evidence, budgets, and authority.

Context is the connective tissue of that stack. Tools produce evidence. Verifiers qualify it. Memory preserves selected lessons. The controller asks for the relevant slice at the moment of action. Without a context layer, every capability grows its own partial memory, retrieval logic, and private version of the truth.

The decisive transition comes when improvement crosses the task boundary. A loop that retries can execute better; a loop that preserves verified lessons can become a better system. A successful trajectory can become memory, memory can become a reusable skill, and a reusable skill can become training data. In that way, yesterday’s search becomes tomorrow’s intuition.

But the trajectory should first become an explicit, verified memory—not merely a transcript of everything that happened. The system can distill the decisive observation, the approaches that failed, the reusable procedure, and the evidence that established success.

trajectory
    ↓ distill + verify
memory
    ↓ reuse + reverify
skill or tool
    ↓ repeated evidence
training signal

Not every lesson should immediately be baked into model weights. Explicit memory is faster to write, easier to inspect, easier to correct, and able to preserve provenance. Lessons that survive repeated reuse can graduate into tools, skills, or training data.

Memory is the bridge between task-local success and lifetime improvement: it lets yesterday’s expensive reasoning alter tomorrow’s starting point. One business implication—still a hypothesis—is that feedback environments may become a durable moat:

Today: “We have the smartest model.”

Tomorrow: “We have five years of verified trajectories showing what actually works.”

Who accumulates that history matters. Model providers may see more aggregate usage, but the enterprise owns the domain-specific outcomes, constraints, and judgments that make those trajectories valuable. Keeping that verified history under enterprise control is what preserves the moat.

The valuable asset would not be raw text alone but accumulated judgment: action → outcome → verification, backed by domain simulators, evaluators, test suites, outcome histories, and trusted review. A competitor can call the same model; it cannot instantly reproduce that history. The system that knows what “good” means may beat one with a slightly smarter generator. It’s in the math !.

---

Beyond loops: self-similar systems#

One final zoom outward: it is tempting to say that graphs are the architectural stage after loops. Graphs are useful because they represent branching workflows, parallel workers, debates, supervisors, specialists, dependencies, and multiple feedback paths.

But a graph is a topology. It does not introduce a new principle of intelligence by itself.

A node in the graph may contain a loop. A team of agents may form another loop. The whole system may then participate in a larger loop with its environment.

Zoom in, and an agent generates, evaluates, and updates. Zoom out, and a team proposes, coordinates, evaluates, and reorganizes. Zoom out again, and an organization observes its environment, allocates agents, acts, measures consequences, and adapts.

The same computational motif appears at several scales.

A verifier may itself be an agent with its own generator, search process, memory, and verifier. A planner may call a collection of specialist loops. A multi-agent system may become one component inside the world model or controller of a still larger system.

The progression may therefore be less like:

loop → graph → more advanced graph

and more like:

loop
  ↓
loops composed from loops
  ↓
coordination between loops
  ↓
self-similar cognitive systems

At that point, coordination becomes a first-class problem. Who gets which task? Who owns shared state? Which conclusions propagate? Who can overrule whom? What should happen in parallel? When should agents communicate? How do we prevent duplicated work? How do we stop one agent’s error from becoming everyone else’s premise? How does a collection of individually competent systems become a competent whole?

These no longer sound like prompting questions. They sound like problems from distributed systems, operating systems, markets, organizations, biology, control theory, and social systems.

Memory becomes more important as these systems grow, not less. It lets agents share state without sharing every token, preserves why decisions were made, and reduces duplicated search. With provenance and write-time checks, it can also help stop one agent’s unsupported conclusion from silently becoming everyone else’s premise.

A graph of amnesiac agents is still amnesia—only in parallel.

---

Why we are building Memseek#

We are building Memseek as an open-source context engine for AI agents. A vector database asks, “What text looks similar?” Memory asks, “What happened before?” A context engine asks:

What should this agent currently believe and use for this decision—and why should we trust it?

That is closer to an epistemic-state engine than a retrieval layer. For each claim, it must maintain the source, supporting evidence, confidence, valid time, contradictions, and the decisions that depend on it.

At Memseek, we believe:

• Claims should be maintained, not merely stored. Evidence stays immutable and source-linked; interpretations remain revisable as new evidence changes what is current.
• More context is not the objective. The objective is the smallest sufficient context for the decision in front of the agent, assembled under an explicit budget.
• Serving context is part of the system, not prompt glue. The sources, selection rules, versions, and exact material served to an agent should be inspectable and reproducible.
• Learning should not mean silent self-rewriting. Outcomes worth learning from should return as evidence, produce an explicit proposal, and pass through verification or review before they change future behavior.

Consider two features we built. A daily briefing could not simply dump the calendar: it had to gather the day’s events, detect unresolved overlaps, and compress what was still relevant into a short morning, afternoon, and evening view. A meeting-time recommender could not simply return empty slots: it had to compare live availability across attendees with working hours and hard constraints, then rank the feasible times using the organizer’s inferred scheduling profile, similar past meetings, adjacent events, attendee time zones, and feedback from rejected suggestions.

These are not separate memory problems. The briefing asks what matters in the schedule now; the recommender asks where a new event belongs in it. They are different views over overlapping evidence and maintained state. Memseek is our attempt to make the rules that produce those views typed, versioned, and reviewable instead of scattering them across prompts, caches, retrieval pipelines, and feature-specific code. We are starting with durable memory because it lets verified experience compound, but the larger goal is a context layer that can observe, maintain, assemble, serve, and learn without becoming opaque.

This blog will follow that work. We will make the pieces concrete: what “current” means, why provenance changes the architecture, how context should be selected and budgeted, how outcomes become learning signals, and where verification belongs when reality does not provide a test suite.

Models will keep improving. Search will make their outputs stronger, verification more trustworthy, and world models will let them investigate consequences before acting. The model generates possibilities; the loop discovers which survive contact with evidence; memory makes those discoveries cumulative.

That is why we care about this project. If the loop has always been there, it should not have to start over.

---

Selected references#

1. Andrej Karpathy, Verifiability. (karpathy.bearblog.dev)
2. Andrej Karpathy, autoresearch. (github.com)
3. DeepMind, AlphaGo Zero: Starting from Scratch. (deepmind.google)
4. DeepMind, FunSearch: Making New Discoveries in Mathematical Sciences Using Large Language Models. (deepmind.google)
5. DeepMind, AlphaEvolve: A Gemini-powered Coding Agent for Designing Advanced Algorithms. (deepmind.google)
6. DeepMind, MuZero: Mastering Go, Chess, Shogi and Atari Without Rules. (deepmind.google)
7. Cheng et al., Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs. (arxiv.org)
8. Gläscher et al., States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning. (pmc.ncbi.nlm.nih.gov)
9. Engeler & Gilbert, The Effect of Metacognitive Training on Confidence and Strategic Reminder Setting. (pmc.ncbi.nlm.nih.gov)
10. Google DeepMind, AlphaZero: Shedding New Light on Chess, Shogi, and Go. (deepmind.google)
11. Google DeepMind, AlphaGo. (deepmind.google)
12. Nassim Nicholas Taleb, An Expert Is Someone Who Knows Exactly What Not to Be Wrong About. (x.com)
13. Chen et al., Evaluating Large Language Models Trained on Code. (arxiv.org)
14. Yao et al., $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. (arxiv.org)
15. Daniel Kahneman, Of 2 Minds: How Fast and Slow Thinking Shape Perception and Choice. (scientificamerican.com)
16. Alex Graves, Adaptive Computation Time for Recurrent Neural Networks. (arxiv.org)
17. Dehghani et al., Universal Transformers. (research.google)
18. Chen et al., Learning to Stop While Learning to Predict. (research.google)
19. Banino, Balaguer & Blundell, PonderNet: Learning to Ponder. (arxiv.org)
20. Rabanser et al., Towards a Science of AI Agent Reliability. (arxiv.org, HAL Reliability Dashboard)
21. METR, Measuring AI Ability to Complete Long Software Tasks. (metr.org, arxiv.org)
22. Aaron Levie, AI agents and the value of exhaustive analysis. (x.com)
23. Norbert Wiener, Cybernetics: Or Control and Communication in the Animal and the Machine. (mitpress.mit.edu)
24. W. Ross Ashby, An Introduction to Cybernetics. (ashby.info)