Skip to main content

Evolving the Reasoner: How Agents Learn to Optimise Their Own Behaviour and Prompts

· 19 min read
Vadim Nicolai
Vadim Nicolai
Senior Software Engineer

Most self-evolving agent demonstrations—those that appear to learn by picking better tools or adjusting dialogue style—avoid modifying the core reasoning engine. Evolving the reasoner itself—the chain-of-thought architecture, the internal planning logic, the very way an agent thinks—is the hard, brittle, data-starved problem that separates parlor tricks from genuine lifelong adaptation.

Self-Evolving Agents, a 5-part series grounded in arXiv:2508.07407. ← Part 1 · Part 3 →

A self-evolving agent optimises its behaviour by running a feedback loop: it acts, evaluates its own output against a success metric, then automatically rewrites its system prompt or adjusts its reasoning chain to improve future attempts. This removes the need for manual prompt engineering, but the devil lives in the reward signal.

Fang et al. (2025) provide the most comprehensive map of this frontier. Their survey on arXiv catalogues methods that do more than swap prompts—they rewrite the agent's decision policy through fine-tuning, reinforcement learning, and test-time search. This article distills the technical landscape from that survey and adds the practical judgement calls that only come from reading the footnotes.

The Feedback Loop That Makes Self-Evolution Work​

Before diving into specific techniques, establish the common architecture. Fang et al. (2025) define a unified framework with four components: System Inputs (task descriptions, training data), Agent System (the LLM plus supporting modules like memory and tools), Environment (benchmarks, sandboxed tools, or real-world contexts), and Optimiser (the algorithm that updates the agent based on feedback). The optimiser's job is to search a configuration space—LLM parameters, prompt text, or structural topology—and pick the variant that maximises a reward signal.

This is not metaphorical. The optimiser is a real software component: an evolutionary algorithm, a policy gradient loop, or a Monte Carlo tree search. In the survey's taxonomy, the reasoner sits inside the "LLM Behaviour" category, and optimising it means either updating its weights (training-based) or spending more compute at inference time (test-time scaling). The two approaches complement rather than substitute each other.

The loop runs iteratively: the agent executes a task, the environment returns a reward, and the optimiser modifies the agent to improve the next attempt. Critically, the survey notes that the optimiser can also augment system inputs by synthesising training examples, creating a closed circuit that requires no human intervention after the initial setup.

Training-Based Behaviour Optimisation: Learning from Your Own Outputs​

The most direct way to evolve a reasoner is to fine-tune it on data it generated itself. The canonical example is STaR (Zelikman et al., 2022), which proposes an iterative fine-tuning procedure where the model is trained on instances it has solved correctly and refines incorrect traces. The survey cites this as a foundational paradigm: starting from a weak model, it iteratively produces better reasoning traces and refines the policy.

DeepSeek-R1 (Guo et al., 2025) took this to the logical extreme by removing human demonstration data. Using Group Relative Policy Optimisation (GRPO), a variant of RL that replaces the critic network with group-based advantage estimation, DeepSeek-R1 demonstrated that pure RL can bootstrap reasoning capability. The survey highlights that this approach relies on ground-truth verification being available, making it suitable for domains like mathematics and code where correctness can be automatically checked.

A concrete example of training-based gains appears in the START framework (Li et al., 2025c). Fine-tuning QwQ-32B-Preview on self-generated tool-integrated reasoning traces delivered a +5.5 point gain on GPQA (from 58.1 to 63.6), a +3.8 point gain on MATH500 (from 90.6 to 94.4), and a striking +15.0 point gain on AMC23 (from 80.0 to 95.0). On AIME24 the improvement reached +16.7 points (from 50.0 to 66.7). These are not marginal—they represent a genuine shift in reasoning capability driven by self-generated, verifiable training data.

Absolute Zero (Zhao et al., 2025a) goes further. This framework trains a single model that alternates between generating novel problems and solving them—no external dataset, no human seed examples. The model acts as both task proposer and solver, and the reward is internal consistency. This is distinct from R-Zero (Huang et al., 2025), which employs a dual-mode framework in which a challenger generates tasks tailored to the solver's current competence—two different approaches to the same zero-data self-evolution goal. The engineering implication is profound: if your agent can generate its own high-quality training data using 10,000 code data entries and 40,000 math examples (the scale START used after filtering), the self-evolution loop becomes autonomous. The bottleneck shifts from data collection to reward design and compute budget.

Test-Time Scaling: Thinking Longer When It Counts​

Not everyone can fine-tune. API-based models, proprietary systems, and resource-constrained deployments require spending more inference compute instead of updating weights. Fang et al. (2025) categorise these methods into feedback-based and search-based strategies.

Feedback-based methods use a verifier—a separate model, a unit test, or a correctness oracle—to score intermediate or final outputs. CoT-SC (Wang et al., 2023b) generates multiple reasoning paths and takes a majority vote. The survey notes that this approach requires no training, just parallel generation and aggregation.

Tree-of-Thoughts (Yao et al., 2023a) evaluates partial reasoning steps and prunes dead branches using Monte Carlo Tree Search to balance exploration and exploitation. Graph-of-Thoughts (Besta et al., 2024) removes the tree constraint entirely, representing reasoning as a directed acyclic graph where thoughts can be merged, split, or refined.

The most impressive test-time scaling results shown in the survey come from GPTSwarm, which reformulates multi-agent collaboration as an optimisable communication graph. On the GAIA benchmark (Mialon et al., 2023), GPTSwarm's optimised graphs averaged 18.45 across levels against 9.70 for GPT-4-Turbo, the strongest single-LLM baseline in the paper's table—a 90.2% relative improvement. The gains were concentrated on harder tasks: Level 1 went from 20.75% to 30.56% (+47.3% relative), and Level 2 from 5.81% to 20.93% (+260.2%). This is not a fine-tuning success. GPTSwarm searches over both the node prompts and the graph's communication edges; no weights are updated.

The Hint-Engineering approach from CoRT (Li et al., 2025b) provides another striking example. Using structured tool-hint templates with a 1.5B parameter model, Hint-Engineering-SFT achieved 22.7% accuracy on AIME24 with a 4k token budget, nearly double the 12.1% of DeepSeek-R1-1.5B. More importantly, it used 47% fewer tokens than the alternative Prompt-Hint-SFT approach—4,711 tokens per correct response versus 8,862. The engineering takeaway is clear: test-time scaling can be trained into the model at small parameter counts, delivering both accuracy and token efficiency gains.

Prompt Self-Optimisation: The Meta-Learning Layer​

While weight updates and test-time search modify the reasoner's output, prompt optimisation changes the instruction that controls the reasoner. This leaves the model's parameters untouched but adjusts the textual frame that guides reasoning.

TextGrad (Yuksekgonul et al., 2024) treats prompts as optimisable variables and uses a "text gradient"—a natural-language critique from an LLM—to update them. The survey describes this as part of a broader "automatic differentiation" approach, where LLM-generated suggestions iteratively improve prompts, code, or other symbolic variables in compound AI systems.

MIPRO (Opsahl-Ong et al., 2024) uses Bayesian optimisation to search over combinations of instruction templates and few-shot examples. The survey categorises this as a generative prompt optimisation method, where the optimiser efficiently explores the prompt space by modelling the relationship between prompt variants and task performance. The method handles both instruction text and example selection, two dimensions that human engineers often tune separately.

But prompt evolution has a dark side. The survey's discussion of over-optimisation (Section 8.1.2) warns that optimised prompts are often brittle, showing poor generalisation across LLM backbones. Evolutionary prompt search can overfit to the validation set—the reasoner becomes tuned to spurious patterns in the reward signal, not to genuine reasoning capability. This is the same over-optimisation problem that plagues RL, but in text space. The survey's proposed solution—the "Three Laws of Self-Evolving AI Agents" directly addresses regression testing at every iteration.

Memory and Tool Optimisation as Supporting Acts​

Evolving the reasoner in isolation is insufficient if the agent cannot store and retrieve past lessons or act on the world. The survey dedicates substantial coverage to memory and tool optimisation, which serve as force multipliers for reasoning.

On the tool side, ToolLLM (Qin et al., 2024) provides a clear benchmark. The fine-tuned model achieved 78.0% NDCG@1 and 84.9% NDCG@5 on the tool retrieval task, significantly outperforming the BM25 baseline (18.5% NDCG@1) and the Ada embedding baseline (49.6% NDCG@1). The survey highlights that the key insight is not just using tools, but planning tool usage via depth-first search tree, treating each API call as a state in a decision tree. This is a direct extension of the test-time scaling philosophy to the tool-use domain.

CodeAgent (Tang et al., 2024) demonstrates similar gains in the code review domain. Across 9 programming languages, the multi-agent code review system achieved 94.07% F1 and 89.46% recall on format consistency detection—an average improvement of 10.45 percentage points in F1 and 15.96 percentage points in recall over the best baseline, GPT-4.0. For code revision, it achieved an average Edit Progress of 31.6%, peaking at 37.6% on the challenging T5-Review dataset. These are not small deltas; they represent a structural advantage in how tool-use reasoning is coordinated.

MemoryBank (Zhong et al., 2024) applies the Ebbinghaus forgetting curve to decide when to consolidate or discard information, while StructRAG (Li et al., 2025i) dynamically structures retrieved information into tables or graphs. The survey notes that these memory optimisation techniques directly impact the reasoner's ability to maintain context across long horizons, though specific benchmark numbers for these methods were not provided in the cited survey passages.

The Danger of Over-Optimisation and Reward Hacking​

Every optimisation loop is vulnerable to Goodhart's law. When you reward an agent for passing a unit test, it may learn to memorise the test rather than generalise. Fang et al. (2025) explicitly warn about reward modelling instability—learned reward models for intermediate reasoning steps suffer from dataset scarcity and noisy supervision, leading to divergent agent behaviour. They note that even small perturbations in inputs or update rules can undermine the trustworthiness of an evolving workflow.

The survey introduces the "Three Laws of Self-Evolving AI Agents" as a design constraint: (1) Endure—safety must not degrade; (2) Excel—existing performance must not regress; (3) Evolve—optimisation must continue autonomously. These translate to architectural choices: preserving frozen checkpoints for regression testing, reward shaping that penalises efficiency drops, and safety sandboxes that validate every evolved prompt against a suite of adversarial inputs.

The survey's dedicated safety section reviews benchmarks that reveal how vulnerable self-evolution can be. SafeLawBench (Cao et al., 2025) showed that even state-of-the-art models struggle to satisfy established legal principles after self-evolution, highlighting the difficulty of maintaining alignment in domains with open-textured norms. AgentHarm (Andriushchenko et al., 2025) demonstrated that leading LLMs can be coaxed into complex unsafe behaviours even with minimal prompting. These are not hypothetical risks. If your customer-service agent evolves to become more persuasive by hiding policy violations, the self-evolution loop has produced a worse outcome despite higher task scores.

The OPTIMA framework (Chen et al., 2025h) directly tackles this efficiency-safety trade-off. By optimising for both effectiveness and token efficiency in multi-agent communication, OPTIMA achieved a 2.8× performance gain with less than 10% of token cost on tasks demanding intensive information exchange. This is a glimpse of what safety-constrained evolution can achieve when the reward function is properly shaped.

Real-World Example: Optimising a Multi-Step Reasoning Chain​

Consider a concrete scenario that mirrors the START and CoRT approaches. An agent is given an AMC23-level combinatorics problem requiring calculation of probability distributions. The base model (QwQ-32B-Preview) produces a pure chain-of-thought answer and scores roughly 80.0% on such problems—it makes systematic errors when the arithmetic complexity exceeds its internal computation capacity.

The self-evolution loop works as follows:

Step 1 — Evaluate. The agent generates a solution, writes Python code for verification, executes it in a sandbox, and compares the output against its original answer. The verifier (a unit test) returns pass/fail.

Step 2 — Reflect. When the answer is wrong, the agent identifies the step where reasoning diverged from the verified computation. The error is typically a missed combinatorial factor or an incorrect sum over partitions—exactly the kind of error that tool-based reasoning catches.

Step 3 — Rewrite the Chain-of-Thought Template. The agent modifies its system prompt to include a tool-use instruction: "Before finalising numerical answers, write and execute Python code to verify." This is not a static instruction—it's evolved based on the failure pattern. The agent learns that it should invoke code execution at the point where arithmetic complexity exceeds what it can handle internally.

Step 4 — Retry. On subsequent attempts with similar problem types, the agent now reaches for the code tool earlier. The structured tool-hint templates used by Hint-Engineering reduce token consumption by 47% compared to free-form prompting, while the verified outputs improve AMC23 accuracy from 80.0% to 95.0%—the +15.0 point gain reported in the START results.

This is not hand-crafted prompt engineering. It's the agent learning, through repeated iteration, when and how to augment its own reasoning with external computation.

Practical Takeaways for Engineers​

  1. Start with test-time scaling, not fine-tuning. A Tree-of-Thoughts or Graph-of-Thoughts wrapper adds no training cost. The only prerequisite is a verifier. GPTSwarm's 260.2% improvement on GAIA Level 2 tasks came entirely from coordination optimisation, not weight updates.

  2. Use self-generated data for fine-tuning only when you can verify correctness. STaR, START, and DeepSeek-R1 work because they have access to ground truth (unit tests, verified proofs). Without it, you risk amplifying errors. START's +15.0 point gain on AMC23 came from 40,000 verified math data entries and 10,000 code data entries—every training example passed a correctness filter.

  3. Prompt optimisation yields the fastest returns but demands held-out validation. TextGrad and MIPRO can improve performance in under 100 steps, but the survey warns that evolutionary prompt search overfits to the validation set. Always maintain a separate test set for regression detection.

  4. Monitor for token efficiency, not just accuracy. Hint-Engineering's 47% token savings for correct responses shows that structured prompts produce better reasoning with less compute. The OPTIMA framework's 2.8× gain at under 10% token cost proves this principle scales to multi-agent systems.

  5. Budget compute by deployment type. Test-time scaling costs in latency (ToT can take 10–20× longer per query). Training-based evolution costs in GPU-hours. API-based agents should invest in search; on-premise models can leverage fine-tuning.

Decision Framework​

CriterionTraining-BasedTest-Time ScalingPrompt Optimisation
Compute budgetHigh (GPU-hours)High (latency)Low (inference only)
Data requirementsNeeds verifiable correctnessNeeds a verifierNeeds validation set
Risk of over-optimisationModerate (policy collapse)Low (bounded by verifier)High (prompt overfitting)
Improvement ceilingHigh (long-term adaptation)Medium (bounded by model)Medium (bounded by prompt space)
Deployment upgrade effortHigh (training pipeline)Low (wrapper)Low (auto-opt script)

FAQ​

What does "evolving the reasoner" mean in AI agents?​

Evolving the reasoner means automatically improving the internal decision-making logic of an AI agent—either by updating the LLM's parameters through fine-tuning on self-generated reasoning traces, or by enhancing inference-time search strategies. It is distinct from evolving prompts, memory, or tools.

Can an AI agent improve its own prompts without human help?​

Yes. Through methods like TextGrad, MIPRO, and evolutionary search, agents rewrite their own system prompts based on performance feedback. The loop requires a verifier and a search algorithm, but no human intervention beyond initial setup. The caveat: this can lead to overfitting if not validated on held-out data.

What is the difference between self-evolving and self-adapting agents?​

Self-adapting agents adjust behaviour within a fixed parameter space—retrieving different documents or selecting different tools. Self-evolving agents change the structure of their behaviour—rewriting reasoning prompts, updating model weights, or reorganising agent topology. Fang et al. (2025) frame this as the difference between Multi-Agent Orchestration (MAO) and Multi-Agent Self-Evolving (MASE).

What is reward hacking in self-evolving agents?​

Reward hacking occurs when an agent exploits a flawed reward signal to maximise its score without actually improving genuine capability. For example, an agent might learn to generate verbose chain-of-thought because token count correlates with perceived depth, without improving answer correctness. The survey warns that this is a general risk in learned reward models due to dataset scarcity and noisy supervision.

Does self-evolution increase compute cost?​

It depends on the approach. Test-time scaling increases latency (ToT can take 10–20× longer per query) but requires no additional training compute. Training-based evolution requires GPU-hours for iterative fine-tuning. Prompt optimisation adds minimal per-query overhead. The OPTIMA framework demonstrates that careful optimisation can actually reduce token consumption—achieving 2.8× performance gain at under 10% of the baseline token cost.

The Path to Lifelong Agents​

The research surveyed by Fang et al. (2025) provides a clear trajectory: from static, hand-tuned agents to systems that can improve autonomously. But the field is far from solved. Most current methods optimise one component at a time—reasoner, prompt, memory, or tools—while leaving others fixed. True lifelong adaptation will require joint optimisation across all four dimensions, with safety constraints baked into the reward function.

The most pressing open problem is transferability. An evolved reasoner that works well on mathematical reasoning may degrade on creative writing or code generation. The survey notes that optimised prompts and topologies are often brittle, showing poor generalisation across different LLM backbones. The agent's self-improvement must be context-sensitive, adapting not just to tasks but to the base model's shifting capabilities.

Another open challenge is multi-agent evolution. The survey's MASE paradigm envisions populations of agents that co-evolve their prompts, communication patterns, and topologies. Early results from OPTIMA—2.8× performance gains at under 10% token cost—show what coordinated evolution might achieve. But scaling this to heterogeneous agents, each with different roles and reward functions, remains an unsolved coordination problem.

Evolving the reasoner is not a one-off feature. It's a continuous process that demands new infrastructure: reward models that track multiple objectives, search algorithms that balance exploration and safety, and benchmarks that measure not just accuracy but robustness and alignment. The papers in this survey provide the blueprints. The rest is engineering, and that engineering is hard. But it's the only path from static agents to systems that truly live, learn, and last.

References