Results

The canonical results for cap-evolve. Every number here is derived from a committed run artifact (examples/*/run_full/*.json) or, where noted, from a held-out run whose artifact is committed separately. The home page's results section is a short snapshot of this page.

Split discipline — how to read every table below
Reward is mean task reward in [0, 1]. Where we quote externally reported results that use 0–100% units, we label them explicitly as percentages. Gains are given as absolute and relative %.

toy_calc — deterministic, zero-API

valtestnotes
Seed prompt0.0—no [CALC] marker
Optimized (mock adds [CALC])—1.0gate-accepted, test sealed

Deterministic, no model call. Asserted by core/tests/test_e2e_slice.py and reproduced by bash examples/toy_calc/run.sh.

τ²-bench airline — no-holdout fit-metric run (reproducible, committed)

Artifact: examples/tau2_airline/run_full/ (final.json, static dashboard under ui/). Reproduce: REPRODUCE_tau2.md.

reward (50 tasks · 10 trials)Δ vs baseline
Baseline (seed policy + tools)0.536—
Best candidate (cand_0007) — val0.712+0.176 / +32.8% relative
cand_0007 — sealed test (fit metric)0.694 pass@1 (pass² 0.584)—

Accepted iterations (the rest were rejected by the gate as within-noise): iter 1 +0.046 (0.536→0.582), iter 3 +0.052 (→0.634), iter 5 +0.036 (→0.670), iter 6 +0.014 (→0.684), iter 7 +0.028 (→0.712). 5 of 10 iterations accepted.

What changed: deep in-code tool edits (tools.py 593 → 832 lines; policy 166 → 233 lines), not just prompt tweaks — five trajectory-verified before→after edits in OPTIMIZATION_EXAMPLES.md; curated walkthrough in examples/tau2_airline/DEMO.md.

τ²-bench airline — held-out 30/10/10 run

Same benchmark and capability, run with a real holdout split (split_ids.json, 30 train / 10 val / 10 test) so the test number is a genuine generalization result.

splitbaselineoptimizedΔ
val (10 tasks)56.770.0+13.3 pp / +23.5% relative
sealed test (10 tasks, scored once)30.047.5+17.5 pp / +58.3% relative

The held-out run_full artifact for this run is committed separately. Until it lands, treat these figures as the reported held-out result; the reproducible artifact-backed run above is the no-holdout fit metric.

See COMPARISON.md for how this +58.3% within-run relative held-out gain sits next to external tool-optimization work (EvoTool, Evolutionary Context Search) — with the important caveat that those use different benchmark versions, models, splits, and budgets and are not an apples-to-apples comparison.

SkillsBench — skill-package optimization (held-out, committed)

Artifact: examples/skillsbench/run_full/ (report.md, final.json). Reproduce: REPRODUCE_skillsbench.md.

rewardΔ
Baseline — val0.333—
Optimized (cand_0004) — val0.714+0.381 / +114% relative
Baseline seed skills — sealed test0.556—
Optimized skills — sealed test (held-out)0.667+0.111 / +20.0% relative

Test was scored once on the sealed split for both baseline and optimized skills, so the improvement is on tasks the optimizer never saw. The optimizer edited all four SKILL.md bodies and added executable scripts, then stopped on a real ceiling (diagnosing two unsolved tasks as broken oracles rather than overfitting them).

At a glance — baseline → optimized

Baseline vs optimized reward across benchmarks Three benchmarks, each showing the seed baseline (open circle) and the optimized candidate (filled green circle) on a 0–100 scale. τ²-bench airline fit metric: 53.6 to 71.2, a gain of 17.6 points. τ²-bench airline held-out test: 30.0 to 47.5, a gain of 17.5 points. SkillsBench held-out test: 55.6 to 66.7, a gain of 11.1 points. 0 25 50 75 100 reward × 100 τ²-bench airline fit metric (50 tasks) 53.6 71.2 +17.6 pp · +32.8% τ²-bench airline held-out test (10 tasks) 30.0 47.5 +17.5 pp · +58.3% SkillsBench held-out test (3 tasks) 55.6 66.7 +11.1 pp · +20.0% baseline (seed) optimized (best candidate)
Rewards normalized to 0–100 for a shared axis. toy_calc is omitted — it's a deterministic loop check (0 → 100 by construction), not a comparative benchmark.

Last reviewed: 2026-07-13.