SkillsBench research · view source on GitHub ↗

SkillsBench task/skill inventory (87 tasks)

Extracted from benchflow-ai/skillsbench@9a1f4dd — each task's task.md frontmatter and environment/skills/ directory.

Category breakdown

Coverage of our current seed skills (docx/pptx/xlsx/pdf)

21 tasks ship one or more of our 4 seed skills in their own environment/skills/. When BenchFlow mounts --skill-mode with-skill --skills-dir <ours>, it replaces the task's own set with ours. So on these tasks the optimization can plausibly matter; on the other 87−21 tasks the office-skill package is out-of-domain.

All 87 tasks, grouped by category

For each task: difficulty — task_type — the skills the task's environment ships (which BenchFlow strips + replaces with the seed when running under --skill-mode with-skill).

software-engineering (16 tasks)

industrial-physical-systems (14 tasks)

office-white-collar (14 tasks)

natural-science (14 tasks)

finance-economics (9 tasks)

mathematics-or-formal-reasoning (8 tasks)

cybersecurity (7 tasks)

media-content-production (5 tasks)

Skill-type (methodological) index

Every task classifies itself by one or more skill_type tags. These describe the kind of skill needed, independent of domain:

Implications for a task-by-task optimization strategy

Two very different setups:

Concrete implications:

1. Skill seed changes per task. For task-by-task, the seed_capability must vary. Adapter's run_batch would run each task with its own skill package as seed. Simplest: for each task, copy environment/skills/* into a per-task seed dir at intake time.

2. Optimizer prompt changes. Currently the optimizer sees N tasks' results and proposes edits to the shared package. For per-task, the optimizer sees just 1 task's results and edits just that task's skill set. Much narrower context; probably faster (fewer tokens) and more focused.

3. Budget scales differently. Full-suite: 43 tasks × 3 trials × 7 iter = 903 rollouts + 7 optimizer calls. Task-by-task: 43 tasks × 3 trials × 7 iter = same 903 rollouts but 43 × 7 = 301 optimizer calls (each smaller). Optimizer $ likely goes up; runner $ same.

4. Comparison with paper. EvoSkills reports 71.1% on all 87 tasks with per-task optimization. Our 43-task subset baseline was 13.95%; best-so-far (after 2 accepted whole-suite iterations) 17.05%. Per-task-EvoSkills-style optimization on our runnable subset would be the direct comparison.

5. Cap-evolve doesn't natively do this. Cap-evolve is designed for one shared skill package. A task-by-task run would either (a) launch 43 separate cap-evolve runs (one per task, each with its own project dir + seed + spec), or (b) require an adapter/algorithm change so cap-evolve loops over tasks within its own iteration structure.

6. The per-task seed does NOT pin the skill vocabulary — verify this per run. The "low out-of-domain risk" claim in the table above assumes each optimization stays within its task's own skills. It isn't enforced: skill-package/scripts/abstract.py discovers sub-packages from disk and permits add, so the optimizer can append a brand-new skill mid-run. Observed on energy-ac-optimal-power-flow, where an invented fuzzy-match/ skill became the accepted champion and the final scored result came out 0.4 → 0.0 — see evidence/energy-ac-optimal-power-flow-vocabulary-violation/, which also documents a second near-miss on simpo-code-reproduction and the mislabelled-trial defect behind the *_CORRUPTED_batch3 runs.