The Four-Component Feedback Loop That Turns a Static Agent Into a Search Problem
Most AI agents you deploy today are frozen the moment they go live. You handcraft the prompts, select the tools, wire up the memory, and hope the configuration survives contact with real users. It doesn't. Tasks drift, APIs change, user intents shift – and your agent silently degrades. The conventional fix is another round of manual reconfiguration. But there's a more principled path: treat agent design not as a one-time assembly but as a continuous search problem.
Self-Evolving Agents, a 5-part series grounded in arXiv:2508.07407. Part 2 →
This isn't a metaphor. In a self-evolving agent, the optimisation loop that refines prompts, tools, memory, and topology is architecturally identical to a search algorithm exploring a state space. The state space is the set of all possible agent configurations. The evaluation function is task performance. The search strategy is the optimiser. The result is an agent that doesn't just execute – it learns.
In this first part of a five-part series, I introduce the four-component feedback loop that makes this possible, grounded in the unified conceptual framework proposed by Fang et al. (2025) in their comprehensive survey of self-evolving AI agents. I'll show why this loop reframes static agent deployment as a search problem, back it with quantitative evidence from frontier systems, and give you a decision framework for choosing when and how to evolve.
What is a self-evolving agent? A self-evolving agent is an AI system that uses a four-component feedback loop—System Inputs, Agent System, Environment, and Optimisers—to iteratively improve its own performance without human intervention, effectively treating each task execution as a search for better configurations. This is the core architectural insight that separates an agent that stores execution history from one that actively searches for better strategies across its own behaviour space.
1. The Static Agent Problem: Why Handcrafted Configurations Fail
Every AI agent today rests on a stack of components: a foundation model (typically an LLM), prompts, memory, tools, and—in multi-agent settings—communication topologies and workflows. In most production systems, these components are set at deployment and never change. The agent runs the same ReAct loop (Yao et al., 2023b) with the same system prompt day after day.
This works perfectly until it doesn't. A customer-service agent handles known intents, but when a new product line launches, its prompts lack context. A code-review agent flags correct patterns as bugs because its few-shot examples are out of date. The core issue is that real-world environments are dynamic, and static agents cannot adapt without human intervention (Fang et al., 2025).
The magnitude of this problem is not theoretical. On the GAIA benchmark, GPTSwarm (Zhuge et al., 2024a) let a multi-agent system optimise itself—jointly searching over node prompts and the edges of its communication graph—rather than fixing that configuration by hand. The optimised swarm scored 30.56% on Level 1 tasks and 20.93% on Level 2, against 20.75% and 5.81% for the strongest single-LLM baseline in the paper (GPT-4-Turbo): relative improvements of 47.3% and 260.2%, and 90.2% on average across levels (18.45 vs 9.70). The gains came from searching the configuration, not from swapping in a better model. A static agent executes instructions; a self-evolving agent searches for better configurations. That search is what separates a brittle deployment from an adaptive system.
2. The Four-Component Feedback Loop: An Overview
Fang et al. (2025) abstract the self-evolving process into four core components that form a closed optimisation loop, illustrated in their conceptual framework (Figure 3 of the survey). The loop flows as follows: System Inputs (task specifications, training data, or input-output pairs) are given to the Agent System (single or multi-agent with prompts, memory, tools, and topology). The agent executes in the Environment, which returns feedback signals via evaluation metrics—accuracy, F1, success rate, or LLM-based proxy scores. These signals feed the Optimiser, which adjusts one or more components of the Agent System—perhaps editing the prompt, pruning a memory entry, or rewiring the communication graph—and the cycle repeats until a target metric is met or convergence is reached.
This is not reinforcement learning over model weights. The optimiser can operate entirely at the inference level, modifying textual prompts, tool descriptions, or workflow structures without fine-tuning the underlying LLM. That distinction is critical for practicality: you can evolve GPT-4o or Claude without access to its parameters.
The framework is governed by what the authors call the Three Laws of Self-Evolving AI Agents (Fang et al., 2025): Endure (maintain safety and stability during modification), Excel (preserve or enhance task performance), and Evolve (autonomously optimise internal components in response to changing tasks). Part 1 focuses on the architecture that enables Evolve, while subsequent parts will address the safety and stability constraints imposed by Endure and Excel.
The marking mechanism is what separates search from history. Without an internal marking heuristic, the loop merely stores execution history. With one—say, a learned value function over past configurations or a process reward model that scores intermediate steps—each execution generates a training signal, converting agent execution into a search over its own behaviour space. This marking is the critical insight: it transforms the agent from a passive recorder into an active explorer of its own configuration landscape.
3. System Inputs: Defining the Search Space
The optimisation loop begins with inputs that scope the problem. Fang et al. (2025) distinguish two levels:
- Task-level optimisation uses a fixed training dataset
Dtrainand a task descriptionT. The goal is to find an agent configuration that generalises across all examples in the held-out test set. - Instance-level optimisation targets a single input-output pair
(x, y). The optimiser tunes the agent for that specific query.
Recent work like Absolute Zero (Zhao et al., 2025a) and R-Zero (Huang et al., 2025) pushes this further by synthesising training data. Absolute Zero trains a single model that alternates between generating problems and solving them, entirely without external data. The model evolves by solving its own puzzles—a fully self-contained loop that eliminates the data bottleneck plaguing most supervised fine-tuning pipelines.
When labelled data is absent, several approaches dynamically synthesise training examples through LLM-based data generation to create surrogate datasets for iterative improvement (Huang et al., 2025; Zhao et al., 2025a; Liu et al., 2025b). Math-Shepherd (Wang et al., 2024d) takes this a step further: instead of relying on outcome-level feedback (pass/fail on final answers), it trains a process reward model (PRM) that scores each intermediate reasoning step. On mathematical reasoning benchmarks, PRM-guided search significantly outperforms outcome-only verifiers because step-level feedback provides dense, actionable signal at every stage of the reasoning chain—not just at the end. The authors demonstrate that this granular marking mechanism enables the optimiser to identify and correct errors mid-trajectory, rather than discarding entire failed reasoning paths. This is a concrete example of the marking mechanism in action: each reasoning step becomes a labelled data point for the search.
4. The Agent System: What's Being Searched
The agent system is the object of optimisation. In a single-agent setup, the search space includes:
- LLM behaviour: reasoning strategies (CoT, ToT, Graph-of-Thoughts) and test-time compute scaling.
- Prompts: instruction phrasing, few-shot examples, and system directives.
- Memory: what to store, how to index, and when to retrieve.
- Tools: which tools to expose, how documentation is formatted, and whether to create new tools.
In multi-agent systems, the search space expands to include topology (who talks to whom) and workflows (the order and logic of agent interactions). The insight from the Fang et al. (2025) survey is that these are not independent. A well-crafted prompt cannot compensate for a poor communication structure, and vice versa. The search must jointly optimise across dimensions.
Take tool optimisation. ToolLLM (Qin et al., 2024) demonstrated that using a depth-first search tree (DFSDT) over tool calls significantly outperforms the standard ReAct approach. On the ToolBench benchmark with over 16,000 real-world APIs, ChatGPT + DFSDT reached an average pass rate of 64.8% and an average win rate of 64.3% against the ChatGPT-ReACT baseline, surpassing even GPT-4 + ReACT in pass rate. Critically, the baseline comparison shows that Vicuna and Alpaca—models with strong general instruction-following—failed to pass any instruction (0% pass rate and 0% win rate), confirming that tool-use capability requires targeted optimisation, not just general language skill. The search space here is not just which tool to call, but the order and backtracking strategy—a direct parallel to classical tree search in AI planning.
CodeAgent (Tang et al., 2024) extends this search to multi-agent code review. On the format consistency detection task (FA), CodeAgent achieved an F1-score of 94.07% and a Recall of 89.46%, representing improvements of 10.45 and 15.96 percentage points respectively over the best baseline (GPT-4.0, which scored 83.62% F1 and 73.50% Recall). On the code revision task, it achieved an average Edit Progress of 31.6% (37.6% on the hardest of the three datasets, T5-Review), compared to average Edit Progress of −1.1% for CodeBERT and −0.1% for CodeT5. These gains come from optimising the collaborative workflow between coder, reviewer, and tester agents—a joint search over both prompts and interaction topology.
5. The Environment: Where Feedback Comes From
The environment is the source of truth for performance. Most benchmarks use task-specific metrics: accuracy, F1, success rate, pass@k. But in real-world deployments, ground truth is often unavailable. Fang et al. (2025) note that many systems now rely on LLM-based evaluators to generate proxy scores or textual feedback when labelled data is absent.
The environment must provide granular feedback to be useful for optimisation. Outcome-level feedback (pass/fail) is sparse. Step-level feedback (process reward models) is denser and generally yields better improvements (Wang et al., 2024d). As described in Section 3, Math-Shepherd's process reward model scores each intermediate reasoning step, enabling the optimiser to correct errors mid-trajectory rather than discarding entire failed paths. This granularity is the difference between knowing that you failed and knowing where you failed.
START (Li et al., 2025c) provides a compelling case for how feedback granularity drives improvement across diverse benchmarks. Using Python tools to enhance long-chain reasoning, START achieved a Pass@1 of 63.6% on GPQA (+5.5pp over QwQ-32B-Preview), 94.4% on MATH500 (+3.8pp), 95.0% on AMC23 (+15.0pp), 66.7% on AIME24 (+16.7pp), and 47.3% on LiveCodeBench (+5.9pp). These improvements come from a two-stage training pipeline: rejection sampling fine-tuning (RFT) on self-generated tool-use trajectories, followed by reinforcement learning with verifiable rewards. The environment here provides both outcome-level correctness (did the code pass the test?) and process-level traceability (did the tool call execute without error?), producing dense feedback that the optimiser uses to refine the model's tool-integrated reasoning policy.
6. The Optimiser: The Search Engine
The optimiser is the engine of the loop. It defines how the search space is explored and which agent configurations are tried. Fang et al. (2025) categorise optimisation algorithms into several families, structured hierarchically: rule-based heuristics (e.g., GRIPS, Prasad et al., 2023), gradient-based methods using text gradients (ProTeGi, TextGrad), Bayesian optimisation and Monte Carlo Tree Search (MCTS), reinforcement learning, and evolutionary strategies (EvoPrompt, Promptbreeder).
The choice of optimiser determines the trade-off between exploration efficiency and solution quality. AFlow (Zhang et al., 2025j) replaces the reinforcement-learning approach of earlier workflow generators (e.g., AutoFlow) with MCTS over typed, reusable operator graphs. AutoFlow searched natural-language programs (CoRE) using RL with both fine-tuning and in-context support, but its exploration was sample-inefficient due to the unstructured search space. AFlow's MCTS with LLM-guided expansion and soft probabilistic selection provides more structured exploration, discovering agentic workflows that outperform handcrafted baselines on coding and reasoning tasks. The ablation is clear: switching from RL over NL programs to MCTS over typed operator graphs yields better sample efficiency without requiring gradient updates to the model.
TextGrad (Yuksekgonul et al., 2024) takes a different approach: it treats textual feedback as a "gradient" and applies automatic differentiation to prompts, code, and other symbolic variables within compound AI systems. The key insight is that text gradients provide a natural-language signal that is interpretable and actionable, unlike numerical gradients over model parameters.
AgentPrune (Zhang et al., 2025g) illustrates how pruning optimises multi-agent communication. It defines the search space as a spatial-temporal communication graph where both intra-dialogue edges (within a single interaction round) and inter-dialogue edges (across rounds) are pruning targets. Using a trainable low-rank-guided graph mask, it performs one-shot pruning to identify and eliminate redundant communications, optimising for token economy. The result is a system that automatically finds the minimal communication topology for a task, reducing inference cost while maintaining accuracy. OPTIMA (Chen et al., 2025, Findings of ACL) pushes this further by directly targeting communication efficiency: as reported in the Fang et al. survey, it achieves a 2.8× performance gain with less than 10% of the token cost on tasks demanding intensive information exchange, through SFT, DPO, and hybrid optimisation of multi-agent communication protocols. Note the scope: this is inter-agent communication on information-exchange-heavy tasks, and it is a trained result—it does not come free of training compute.
7. Why This Feedback Loop Converts Agent Execution Into a Search Problem
Here is the core insight: without the optimiser, the agent is a deterministic function from input to output. With the optimiser in the loop, each execution of the agent becomes a data point for the search. The agent no longer simply solves the task; it also generates information about which configurations work. The optimisation loop treats the space of possible agent systems (prompts, tools, memory, topology) as the state space, and the performance score as the evaluation function.
This is directly analogous to classical optimisation in AI: the search over a decision tree, the exploration of a hyperparameter grid, or the evolution of a genetic algorithm population. The difference is that the "genome" is not fixed-length bits but natural-language prompts, tool descriptions, and workflow code. The search operators are not crossover and mutation but text editing, LLM-based generation, and graph pruning.
Fang et al. (2025) formalise this as: A* = argmax_{A in S} O(A; I), where S is the search space of agent configurations, O is the evaluation function, and A* is the optimal agent. The optimiser's job is to navigate S efficiently. This reframing unites many disparate techniques—prompt engineering, memory management, tool selection—under a single mathematical lens.
Without an internal marking heuristic, the loop merely stores history. With one—say, a learned value function over past configurations—each execution generates a training signal, converting agent execution into a search over its own behaviour space. AgentPrune (Zhang et al., 2025g) embodies this exactly: its trainable graph mask learns which communication edges are valuable, marking some for retention and others for pruning. That marking is what turns topology design from a manual choice into an optimisable search variable. The marking mechanism is the critical insight that differentiates an agent that "remembers what happened" from one that "learns what to try next."
8. Quantitative Evidence from the Frontier
The table below consolidates the quantitative claims across the research landscape:
| Technique | Metric | Performance | Source |
|---|---|---|---|
| GPTSwarm (GAIA L1) | Accuracy | 20.75% → 30.56% (+47.3% rel. vs GPT-4-Turbo) | Zhuge et al., 2024a |
| GPTSwarm (GAIA L2) | Accuracy | 5.81% → 20.93% (+260.2%) | Zhuge et al., 2024a |
| GPTSwarm (GAIA avg) | Accuracy | 9.70% → 18.45% (+90.2%) | Zhuge et al., 2024a |
| ToolLLM (ChatGPT+DFSDT) | Win rate vs ReACT | 64.3% (pass rate 64.8%) | Qin et al., 2024 |
| CodeAgent (FA task) | F1-score | 94.07% (+10.45 pp over GPT-4.0) | Tang et al., 2024 |
| CodeAgent (FA task) | Recall | 89.46% (+15.96 pp over GPT-4.0) | Tang et al., 2024 |
| CodeAgent (CR task) | Edit Progress | 31.6% avg | Tang et al., 2024 |
| START (GPQA) | Pass@1 | 63.6% (+5.5pp) | Li et al., 2025c |
| START (MATH500) | Pass@1 | 94.4% (+3.8pp) | Li et al., 2025c |
| START (AMC23) | Pass@1 | 95.0% (+15.0pp) | Li et al., 2025c |
| START (AIME24) | Pass@1 | 66.7% (+16.7pp) | Li et al., 2025c |
| START (LiveCodeBench) | Pass@1 | 47.3% (+5.9pp) | Li et al., 2025c |
| OPTIMA (inter-agent comms) | Perf gain vs cost | 2.8× gain, <10% tokens | Chen et al., 2025h |
| AFlow (MCTS search) | Pass@k | Outperforms handcrafted | Zhang et al., 2025j |
Several patterns emerge:
-
The improvement is largest where manual design is hardest: multi-agent communication graphs (GPTSwarm's 260% Level 2 gain) and complex tool chains (ToolLLM's DFSDT) benefit most from automated search.
-
Search-based methods consistently outperform one-shot optimisation: AFlow and GPTSwarm both use iterative search (MCTS and RL respectively) and beat their non-searched baselines by wide margins.
-
The optimiser matters more than the base model: the same underlying LLM (ChatGPT) with a better search strategy (DFSDT) surpasses GPT-4 with a weaker one (ReACT) in pass rate. START's 16.7pp gain on AIME24 comes from optimising tool-use trajectories, not from a larger model.
9. Decision Framework: When and How to Build a Self-Evolving Agent
Not every agent needs self-evolution. Use this framework to decide:
- If your environment is static (fixed APIs, known user intents, stable tasks) → Stay static. The overhead of the loop is not justified.
- If your environment changes slowly (quarterly product updates, seasonal shifts) → Use scheduled evolution (batch optimisation offline, then redeploy).
- If your environment is dynamic and unpredictable (new tools daily, user intents shift hourly) → Build the full online loop with real-time feedback and periodic re-optimisation.
- If performance degrades over time due to drift → Add an optimiser that monitors drift and triggers automatic re-optimisation.
- If your task is multi-agent with complex coordination → Self-evolution is almost certainly necessary. Manual topology design is brittle at scale.
The most practical entry point is to start with prompt optimisation (using OPRO, TextGrad, or EvoPrompt) on a fixed agent architecture. This requires minimal changes to your existing system and needs no model fine-tuning. Once that's stable, add memory optimisation and tool selection. Only then graduate to topology and workflow evolution.
How does a static agent differ from a self-evolving agent? A static agent executes a fixed configuration across all interactions—it receives input, runs its preset prompts and tools, and produces output, degrading silently as its environment changes. A self-evolving agent closes the loop: it observes its own performance, evaluates it against a metric, marks successful and failed configurations in memory, and adjusts its future behaviour accordingly. The static agent is a function; the self-evolving agent is a search algorithm optimising that function over time.
Closing: The Paradigm Shift from Deployment to Continuous Search
The four-component feedback loop does more than improve agent performance. It changes how we think about deploying AI systems. Instead of shipping a frozen artifact, you ship a search algorithm that continuously refines its own configuration. The agent becomes a participant in its own optimisation, not a passive executor.
This is the first step in a five-part series. In Part 2, I will dive deep into prompt optimisation—the most accessible and immediately useful form of self-evolution. You'll see how edit-based, generative, and text-gradient methods compare, and how to choose the right one for your use case. By the end of the series, you'll have a complete toolkit for building agents that endure, excel, and evolve—the Three Laws of Self-Evolving AI Agents (Fang et al., 2025: Endure for safety adaptation, Excel for performance preservation, and Evolve for autonomous optimisation).
The evidence is clear: static agents leave performance on the table. The question is not whether to evolve, but how. The four-component loop gives you the architecture. The research gives you the algorithms. Now it's time to deploy.
References cited: Fang et al. (2025), Zhuge et al. (2024a), Qin et al. (2024), Tang et al. (2024), Li et al. (2025c), Yuksekgonul et al. (2024), Zhang et al. (2025g), Zhang et al. (2025j), Zhao et al. (2025a), Huang et al. (2025), Wang et al. (2024d), Prasad et al. (2023), Feng et al. (2025a), Guo et al. (2024b), Fernando et al. (2024), Chen et al. (2025h).