SkillsBench task-by-task-87 — full-benchmark heatmap

87 tasks · dedicated LSF host per task (CCC) or dedicated Docker container (Mac) · 10 trials per candidate · up to 4-iteration budget · Sonnet-5 agent + Opus-4.8 optimizer. Data merged from four sources. Badges — source: c3v1 (CCC, bucket-1/2/3 unblocks, freshest) · c2v2 (CCC, 6 clean reruns) · c1v5 (CCC, post-cache-clear) · c1v1 (CCC, original 43) · c1tA (CCC, Track A) · mac-v2 (Boaz's Mac, 7 CCC-blocked retried); state: no-signal (task scored 0.0 for both baseline and every candidate — cap-evolve had no lever).

Click any task name to open its report — what changed, what worked, what didn't, and what we can (or can't) learn from that task. See reports/README.md for how far along that is: tasks already at 1.0 on seed, and the four no-signal tasks that are infrastructure failures, are answered in full; most of the rest are still placeholders.

Cells: mean reward across 10 trials on the val split. Grey/"—" = candidate not run.
Task names link to per-task reports in reports/task-by-task/.
Best (amber): best mean across all evaluated candidates; the best cell has a purple ring.
Amber-bordered baseline: saturated at seed (val=1.0); optimizer skipped.
no-signal badge: task scored 0.0 for both the seed and every optimizer candidate — cap-evolve had no signal to work with. 10 tasks in this state: 4 where the container built but the agent produced empty tool_calls (earthquake-phase-association, fix-build-google-auto, fix-visual-stability, seismic-phase-picking), and 6 where cap-evolve ran to completion but no candidate moved off zero (financial-modeling-qa, fix-build-agentops, fix-druid-loophole-cve, python-scala-translation, quantum-numerical-simulation, spring-boot-jakarta-migration).

c4 re-run — prev (c1/c2/c3) vs current (c4)

10 tasks re-run in a fresh worktree (intake_skillbench_c4) against a from-scratch seed capability, up to 6-iteration budget, optimizer model claude-opus-4-6 (down from claude-opus-4-8 on the prior run of these same tasks). "prev" is each task's own row from the full-87 heatmap above (same source badges apply); "curr" is the c4 run. cap-evolve's optimizer state does not carry forward between independent runs against a shared skill — a fresh run has to re-derive any fix from scratch within its budget, so a plateau here is lost progress, not a new bug. See evidence/shock-analysis-demand-optimizer-regression for a fully-worked example (shock-analysis-demand, Δ +0.9 → +0.0).

Two rows per task: prev = prior run's val scores (same source badge/data as the full-87 heatmap); curr = this c4 run's val scores. Grey/"—" = candidate not run.
Best (amber), purple ring: as above — best mean across all evaluated candidates in that run.
DONE all 10 c4 tasks finished their optimizer budget; Best is the val leader across seed + all evaluated candidates.
Red ring on curr-best: curr's best fell short of prev's best. Green ring: curr's best beat prev's best.
test column: shown only where finalize sealed a held-out score. organize-messy-files and setup-fuzzing-py hit cap-evolve's TestSealError (an earlier, unrelated finalize attempt had partially scored test without committing a seal, and a second look at test is refused by design) — Best there is the val result, same as every other task. Since train/val/test all hold the identical single task in these single-task runs, test carries no held-out signal anyway; val is the number that matters.
shock-analysis-supply's curr Best (cand_0003=0.667) is the accepted champion from its JOURNAL.md; cand_0004-0006 are independently-rejected candidates whose own scores are noise/failed experiments, not a decline off 0.667. flink-query's curr test (0.0) reflects test-infra noise (Maven/podman failures) per its own JOURNAL.md, not a real signal.