Reflective optimization for agents
Make your agent reliably better — and prove it
Optimize agentic capabilities — with agents.
cap-evolve improves an AI agent's prompts, tools, and skills by learning from its failed evaluation traces. Bring the agent and the eval you already have. cap-evolve runs the loop — evaluate → diagnose → propose an edit → keep it only if it beats a held-out split by a significant margin → commit — and reports one honest number. It optimizes what your agent reads, not its weights.
Try it in two minutesno API key
git clone https://github.com/skillberry-ai/cap-evolve.git
cd cap-evolve
python3 -m venv .venv && source .venv/bin/activate
pip install ./core
bash examples/toy_calc/run.sh
toy_calc is a deterministic zero-API stand-in: the seed prompt scores
0.0; the optimized prompt is gate-accepted and scores 1.0
on the sealed test split — no model is called. Full walkthrough:
Getting started · then
run it on a real benchmark.
How the loop works
One iteration. A coding agent reads the failing traces and proposes a bold, multi-part edit; the framework keeps it only if it clears a held-out significance gate.
Score on val
Run the candidate on the validation split — every trial its own seed for honest variance.
Read the failures
Cluster failing rollouts into a reflective dataset: inputs, outputs, and gold-safe feedback.
Edit the capability
The optimizer agent writes one bold, multi-part change to the prompt, tools, or skill.
Keep only if it wins
Accept only when Δ > k·SE on val. Noise-level gains are rejected and remembered.
Git-commit + learn
Every iteration is a commit; a journal records what worked so the next one builds on it.
What that produces
τ²-bench airline, policy + tool code jointly optimized over 50 tasks × 10 trials (fit metric — no holdout on this committed run).
Most of the lift is real code the optimizer wrote into the tool bodies — deterministic guards, not just prose. See every accepted edit on Results, or reproduce it step by step.
See the whole run — offline
Every run ships a self-contained dashboard export. No backend, no live model — open it and inspect exactly what happened, honestly.
Why cap-evolve
Five things that shape how it works — and how it differs from what came before.
The optimizer is a coding-agent CLI (Claude Code, Codex, and 12 more): it reads failing traces and proposes bold multi-part edits — no gradient descent, no heuristic search.
How the optimizer runs → 02System prompts, tool code, MCP surfaces, skill packages — one at a time or combined. What your agent reads, not its weights.
What it optimizes → 03Cap the cost of each iteration — the optimizer stops at the budget you set, not at the point of diminishing returns. Wall-clock next.
On the roadmap → 04Iteration state lives in git-committed markdown — LEDGER, JOURNAL, PROCESS. Every run is a browseable history you can inspect, edit, or resume.
Test split sealed until finalize, scored exactly once; every accepted iteration clears a paired-significance gate on val — on the primary metric only. Every published number ships its run artifact.
Splits, gates, sealing →Results
Cross-checked against committed run artifacts. Each is labeled fit metric (no holdout) or held-out (test scored once on ids the optimizer never saw). Full detail, models, task/trial counts, and costs: Results.
| Benchmark | Split | Baseline → Optimized | Gain |
|---|---|---|---|
| toy_calc (zero-API) | sealed test | 0.0 → 1.0 |
deterministic proof |
| τ²-bench airline (policy + tools) | val — fit metric | 0.536 → 0.712 |
+0.176 / +32.8% |
| τ²-bench airline (held-out 30/10/10) | sealed test | 30.0 → 47.5 |
+58.3% · reported |
| SkillsBench (skill package) | sealed test (held-out) | 0.556 → 0.667 |
+0.111 / +20.0% |
What cap-evolve optimizes
Pick one — or combine them, e.g. [system-prompt, tools].
| Capability | What the optimizer may change |
|---|---|
| System prompts | Rewrite / consolidate / add rules, examples, output contracts — never drop a needed rule |
| Tool implementations | Edit tool code for deterministic enforcement; add/wrap/swap tools (never bare-remove) |
| MCP tool surfaces | Safe edits only — tool docs, in-description examples, and which tools are exposed |
| Skill packages | An Agent Skill dir — SKILL.md bodies, references, and executable scripts |
Choose your path
Every path shares the same core install and the same honesty guarantees.
| Path | Use it when | Start |
|---|---|---|
| Claude Code plugin | You use Claude Code and want slash commands + honesty hooks | claude --plugin-dir ./plugins/cap-evolve |
| Another coding-agent host | Codex, Gemini, opencode, Cursor, Droid, Copilot, Kimi, Pi, Antigravity, openclaw, IBM Bob, or bare | ./install.sh --host <name> |
| Manual adapter + CLI | You want to wire the adapter yourself and drive cap-evolve directly | Optimize your own agent |
Documentation
Hand-crafted pages for the hero topics; the rest are on GitHub, alongside the code they describe.
Your first successful run in two minutes, no API key. The toy_calc loop end-to-end.
A real benchmark, start to finish: τ²-bench airline from setup to sealed result, step by step.
Full benchmark detail: toy_calc, τ²-bench airline (fit + held-out), SkillsBench. Reproducibility and artifacts.
The raw CI log: every ci/benchmarks execution on PRs and manual runs, sortable and filterable.
The pipeline (intake → check → baseline → algorithm → finalize → report), the optimizer context, and the skill library.
Deterministic vs agent orchestration, and the agent-optimize algorithm — the conversational agent drives the loop itself.
Wire one small adapter — three methods. Two ways: let your coding agent do it, or drive the CLI yourself.
Copy-and-run adapters — JSONL, HuggingFace, tau2, SWE-bench, SkillsBench — for any litellm provider. Config, not code.
Splits, gates, sealing — the discipline that keeps the numbers meaningful.
The full method signatures and semantics if you're implementing an adapter.
Positioning vs other tool-optimization work (EvoTool, Evolutionary Context Search) — with the caveats.
Installation or a run failed — start here.