Results
The canonical results for cap-evolve. Every number here is derived from a committed run
artifact (examples/*/run_full/*.json) or, where noted, from a held-out run whose
artifact is committed separately. The home page's results section is a short
snapshot of this page.
- fit metric —
train == val == test(no holdout); the test number is not held out and the engine logs asplits_warning. Useful to show the loop works; not a generalization claim. - held-out — test ids the optimizer never saw, scored exactly once at
finalize.
Reward is mean task reward in [0, 1]; where a figure is also useful as a percentage we
show both (e.g. 0.536 (53.6%)). Gains are given as absolute
and relative %.
At a glance — baseline → optimized
toy_calc is omitted — it's a
deterministic loop check (0 → 100 by construction), not a comparative benchmark. The held-out
τ² row is the reported (artifact-pending) result.
RH-SWE-bench — Harbor SWE-bench Verified (fit metric)
cap-evolve optimizing a coding agent's system prompt + skill on
SWE-bench Verified via
Harbor for sandboxed evaluation.
Each task runs a full claude-code agent inside an isolated Docker container.
- Capability: agent skill-package + system-prompt (
[skill-package, system-prompt]). - Optimizer:
claude-code@claude-opus-4-6. - Agent under test:
claude-sonnet-4-6via Harbor in Docker containers. - Tasks: 119 val tasks from SWE-bench Verified.
- Split:
train == val == test == 119— fit metric (no holdout). - Algorithm / gate:
hill-climb --focus all, 7 iterations, paired significance gatek_se 1.0.
| reward (119 tasks) | Δ vs baseline | |
|---|---|---|
| Baseline (seed prompt + skill) | 0.580 (58.0%) | — |
Best candidate (cand_0002) — val | 0.765 (76.5%) | +0.185 / +31.9% relative |
2 of 7 iterations accepted: iter 1 +0.118 (0.580→0.698),
iter 2 +0.067 (→0.765). Per-task: 24 improved, 2 regressed, 93 unchanged.
Optimizer spend: $60.75 over ~82 minutes.
These are a different measurement from the 58.0% → 76.5% fit-metric run described above, and the relationship between the two pairs is unresolved — the chart contains neither number.
Honest reading: because
train == val == test, the reward is a fit metric, not a generalization claim. The lift comes from the optimizer editing the coding agent's system prompt and skill to improve how it diagnoses and patches open-source bugs. Harbor provides sandboxed, reproducible evaluation — each task runs the agent in its own Docker container against the repo's test suite.
toy_calc — deterministic, zero-API
| val | test | notes | |
|---|---|---|---|
| Seed prompt | 0.0 | — | no [CALC] marker |
Optimized (mock adds [CALC]) | — | 1.0 | gate-accepted, test sealed |
Deterministic, no model call. Asserted by core/tests/test_e2e_slice.py and
reproduced by bash examples/toy_calc/run.sh.
Honest reading: this is a loop self-check (0.0 → 1.0 by construction), not a comparative benchmark. It proves the accept/gate/seal machinery works end-to-end — nothing more.
τ²-bench airline — no-holdout fit-metric run (reproducible, committed)
Artifact: examples/tau2_airline/run_full/
(final.json, static dashboard under ui/).
Reproduce: REPRODUCE_tau2.md.
- Capability: airline policy + tools optimized jointly (
[system-prompt, tools]). - Optimizer:
claude-code@claude-opus-4-6. - Runner + user simulator:
openai/gpt-oss-120bvia IBM RITS. - Tasks / trials: all 50 airline tasks · 10 trials each.
- Split:
train == val == test == 50— fit metric (no holdout). - Algorithm / gate:
hill-climb --focus all, 10 iterations, paired significance gatek_se 0.2. - tau2-bench commit:
8ebb7499622fc2be9b9d510d6f7a7653461f4f29.
| reward (50 tasks · 10 trials) | Δ vs baseline | |
|---|---|---|
| Baseline (seed policy + tools) | 0.536 (53.6%) | — |
Best candidate (cand_0007) — val | 0.712 (71.2%) | +0.176 / +32.8% relative |
cand_0007 — sealed test (fit metric) | 0.694 (69.4%) pass@1 · pass² 0.584 (58.4%) | — |
Accepted iterations (the rest were rejected by the gate as within-noise):
iter 1 +0.046 (0.536→0.582), iter 3 +0.052 (→0.634),
iter 5 +0.036 (→0.670), iter 6 +0.014 (→0.684),
iter 7 +0.028 (→0.712). 5 of 10 iterations accepted.
What changed: deep in-code tool edits (tools.py 593 → 832 lines;
policy 166 → 233 lines), not just prompt tweaks — five trajectory-verified
before→after edits in
OPTIMIZATION_EXAMPLES.md;
curated walkthrough in
examples/tau2_airline/DEMO.md.
Honest reading: because
train == val == test, the sealed-test number is a fit, not a generalization claim — the engine logs asplits_warning. This run's value is that it is fully committed and reproducible, and that the lift came from real in-code tool edits.
τ²-bench airline — held-out 30(=val)/20 run
Same benchmark and capability, run with a real holdout split
(split_ids.json, train=val=30, test=20) so the test number is a genuine
generalization result.
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (30 tasks) | 0.567 (56.7%) | 0.700 (70.0%) | +13.3 pp / +23.5% relative |
| sealed test (20 tasks, scored once) | 0.300 (30.0%) | 0.475 (47.5%) | +17.5 pp / +58.3% relative |
These held-out figures are reported: the run_full artifact for
this run is not yet committed. Until it lands, treat the numbers above as the
reported held-out result — not artifact-backed. The reproducible,
artifact-backed run is the no-holdout fit metric above.
See COMPARISON.md for how this +58.3% within-run relative held-out gain sits next to external tool-optimization work (EvoTool, Evolutionary Context Search) — with the important caveat that those use different benchmark versions, models, splits, and budgets and are not an apples-to-apples comparison.
τ²-bench airline — agent orchestration mode (held-out 30 (=val) / 20)
The first run driven entirely in agent mode with the
agent-optimize algorithm: the conversational agent understood the benchmark, ran the
baseline, then proposed the airline-policy edits itself, gated every candidate on the full val
split, and sealed the test once. Runner + user simulator: aws/gpt-oss-120b via the
IBM ete litellm proxy. Split: 30 train == 30 val (fit) · 20 held-out test.
| split (single-trial) | baseline | best (cand_5) | Δ |
|---|---|---|---|
| val (30, fit) | 0.500 (50.0%) | 0.633 (63.3%) | +0.133 / +26.7% relative — gate-significant |
| sealed test (20, held-out) | 0.400 (40.0%) | 0.550 (55.0%) | +0.150 / +37.5% relative |
Honest reading: τ²-bench airline is high-variance at a single trial. On a stable 3-trial re-evaluation the val fit gain is +18.4% (paired-significant, Δ/SE = 1.80) while the held-out test is flat within noise (−0.067, SE 0.105) — and because
train == val, the val figure is a fit, not generalization. Agent mode produced the only gate-accepted improvement in a head-to-head against a bounded deterministic run (whose best stayed the seed at sealed test 0.35), butgpt-oss-120b's stable val ceiling here (~0.64) is below the requested 0.78 target; a genuine held-out gain needs a stronger runner model or tool-level edits. Details: Agent mode Harbor.
τ²-bench airline — Qwen 2.5 14B
Same benchmark, capability, split, and algorithm as the held-out run above, with a self-hosted open model (Qwen 2.5 14B-Instruct via vLLM on OpenShift) replacing the Claude runner.
- Capability: airline policy + tools (
[system-prompt, tools]). - Optimizer:
claude-code@claude-sonnet-4-6(Vertex AI). - Runner + user simulator:
Qwen/Qwen2.5-14B-Instructvia vLLM on OpenShift. - Tasks / trials: 50 airline tasks · 5 trials each.
- Split: train=val=30, test=20 — held-out.
- Algorithm / gate:
hill-climb --focus all, 10 iterations, paired significance gatek_se 0.3.
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (30 tasks) | 0.200 (20.0%) | 0.387 (38.7%) | +0.187 / +93.5% relative |
| sealed test (20 tasks, scored once) | 0.170 (17.0%) | 0.240 (24.0%) | +0.070 / +41.2% relative |
3 of 10 iterations accepted. The optimizer added code-level enforcement of cancellation rules, input validation guards, and pre-flight status checks — hardening tool implementations rather than rewriting policy prose.
τ²-bench airline — Qwen 2.5 14B, all capabilities
Same model and split as above, with all three capability types optimized jointly.
- Capability:
[skill-package, system-prompt, tools]. - Optimizer:
claude-code@claude-sonnet-4-6(Vertex AI). - Runner + user simulator:
Qwen/Qwen2.5-14B-Instructvia vLLM on OpenShift. - Tasks / trials: 50 airline tasks · 5 trials each.
- Split: train=val=30, test=20 — held-out.
- Algorithm / gate:
hill-climb --focus all, 10 iterations, paired significance gatek_se 0.3.
| split | baseline | optimized | Δ |
|---|---|---|---|
| val (30 tasks) | 0.273 (27.3%) | 0.520 (52.0%) | +0.247 / +90.5% relative |
| sealed test (20 tasks, scored once) | 0.120 (12.0%) | 0.270 (27.0%) | +0.150 / +125.0% relative |
3 of 10 iterations accepted, 7 rejected. The optimizer edited tool code, policy rules, and skill prose jointly — combining code-level guards with policy clarifications and structured methodology in SKILL.md. Best held-out test gain across all Qwen 14B runs (+125%).
Key takeaway — Qwen 14B runs: On a self-hosted 14B model,
[skill-package, system-prompt, tools]withhill-climbproduced the best held-out result (+125% on test). Joint optimization of all three capabilities outperformed the tools-only run (+41.2%) when paired with hill-climb's conservative gating.
RH-SWE-bench — Harbor SWE-bench Verified (fit metric)
cap-evolve optimizing a coding agent's system prompt + skill on
SWE-bench Verified via
Harbor for sandboxed evaluation.
Each task runs a full claude-code agent inside an isolated Docker container.
- Capability: agent skill-package + system-prompt (
[skill-package, system-prompt]). - Optimizer:
claude-code@claude-opus-4-6. - Agent under test:
claude-sonnet-4-6via Harbor in Docker containers. - Tasks: 119 val tasks from SWE-bench Verified.
- Split:
train == val == test == 119— fit metric (no holdout). - Algorithm / gate:
hill-climb --focus all, 7 iterations, paired significance gatek_se 1.0.
| reward (119 tasks) | Δ vs baseline | |
|---|---|---|
| Baseline (seed prompt + skill) | 0.580 (58.0%) | — |
Best candidate (cand_0002) — val | 0.765 (76.5%) | +0.185 / +31.9% relative |
2 of 7 iterations accepted: iter 1 +0.118 (0.580→0.698),
iter 2 +0.067 (→0.765). Per-task: 24 improved, 2 regressed, 93 unchanged.
Optimizer spend: $60.75 over ~82 minutes.
SkillsBench — skill-package optimization (held-out, committed)
Artifact: examples/skillsbench/run_full/
(report.md, final.json).
Reproduce: REPRODUCE_skillsbench.md.
- Capability: the four shared office-document skill packages (
docx/pptx/xlsx/pdf). - Agent under test:
claude-sonnet-4-6in a Docker sandbox. - Optimizer:
claude-code@claude-opus-4-8. - Tasks / trials: 7 val tasks (
train == val) · 3 trials; 3 sealed test tasks. - Iterations: 7 (best
cand_0004, 4 accepted).
| reward | Δ | |
|---|---|---|
| Baseline — val | 0.333 (33.3%) | — |
Optimized (cand_0004) — val | 0.714 (71.4%) | +0.381 / +114% relative |
| Baseline seed skills — sealed test | 0.556 (55.6%) | — |
| Optimized skills — sealed test (held-out) | 0.667 (66.7%) | +0.111 / +20.0% relative |
Test was scored once on the sealed split for both baseline and optimized
skills, so the improvement is on tasks the optimizer never saw. The optimizer edited all
four SKILL.md bodies and added executable scripts, then stopped on a real
ceiling (diagnosing two unsolved tasks as broken oracles rather than overfitting them).
Honest reading: this is a committed, held-out result — the test split was scored once for both baseline and optimized skills. The val jump (+114%) is large but on 7 fit tasks; the sealed-test gain (+20.0%) is the honest generalization number.
Last reviewed: 2026-07-24.