Optimizing using the blackbox approach
In the blackbox approach the optimizer process never touches the benchmark.
Introduction
Some clients, such as enterprises, cannot share their agent's code, sources or behaviour - with an external optimization process. For them the optimization runs indirectly, without inspecting or directly modifying their agents.
The approach introduces two additional services:
- Skillberry Store: a smart repository of tools and skills for agentic workflows.
- Skillberry Proxy-Agent: a proxy that sits between the client agent and the LLM provider, uses the optimized skills and tools, and returns an optimized response to the agent.
In the blackbox approach, cap-evolve deploys the capabilities to optimize (system prompts, tools, skills) into the Skillberry Store, and the Skillberry Proxy-Agent reads them and injects them into the agent's LLM calls. The benchmark agent is left unchanged; it receives the optimized response from Skillberry.
The optimized capabilities can be used in two ways:
- Through the Skillberry service: the client never changes its benchmark. The benchmark is optimized through the responses the Skillberry Proxy-Agent returns.
- By exporting the optimized capabilities from the Skillberry Store and patching them back into the client's benchmark.
How it works
The diagram above shows the flow between the benchmark agent and the real model it reaches. The agent receives an optimized response without being modified.
Setup
Assumes cap-evolve is already installed (getting started)
and your benchmark's adapter is in .capevolve/project/adapters/.
# 1. Provision the Skillberry Store and Proxy-Agent (clones, venvs, installs)
python -c "import sys; sys.path.insert(0, 'skills/interventions/llm-proxies/blackbox/scripts'); \
import blackbox_env; blackbox_env.provision()"
# 2. Check the stack is ready
python skills/interventions/llm-proxies/blackbox/scripts/run.py
This clones both services under vendor/ and reports, for each one, whether it is
provisioned, running and healthy. A run never provisions anything itself: its preflight stops
if the stack is not up.
What your benchmark must provide. The blackbox approach checks these, but does not supply them:
- The agent's LLM calls go to the Proxy-Agent. Other calls (user simulator, judges, verifiers) go straight to the model.
- The benchmark's environment is reachable over HTTP, because the skill's tools run in the Store, not where the benchmark's state lives.
- The adapter deploys each candidate in
apply(): it makes the candidate the one skill in the Store, then restarts the Proxy-Agent on it.
Configure and run
# What the blackbox approach runs
SPA_MODEL_NAME=<model> # the model the Proxy-Agent calls on the agent's behalf
SPA_REMOTE_ENV_URL=<url> # where your benchmark's environment service listens
SKILLBERRY_STORE_PORT=8000 # the Store's port
SKILLBERRY_AGENT_PORT=7000 # the Proxy-Agent's port
# capevolve.yaml — the keys that select the approach
intervention: blackbox
skill_name: <skill> # required: the one skill the Proxy-Agent serves
# what to optimize
capabilities: [tools] # or [system-prompt], [skill-package], ...
capability_path: seed_capability # the seed; a copy is optimized each iteration
# Run the optimization
cap-evolve run --spec .capevolve/project/capevolve.yaml
Each iteration, cap-evolve deploys the current candidate to the Skillberry Store, and the
Proxy-Agent injects it into the agent's LLM calls.
cap-evolve then collects the rewards, gates the edit and repeats. The benchmark itself is never modified.
Output lands in .capevolve/run_<timestamp>/, with report.md,
dashboard.html and the optimized candidates.
The run's Config tab shows intervention: blackbox under
Delivery.
An onboarding example (τ²-bench)
A coding agent onboards a benchmark for the blackbox approach by following the prompt below and
answering blackbox when it asks which approach to use. It installs the benchmark
unmodified, seeds the capability from the benchmark's own source, and adds the tailoring the
approach needs from outside the benchmark. It then passes the cap-evolve check gate
and runs the optimization loop: it proposes edits, keeps only those that clear the significance
gate, scores the sealed test once, and writes a report.
Reference:
benchmarks/tau2/examples/sierra_airline/PROMPT.md