Optimizing using the blackbox approach

In the blackbox approach the optimizer process never touches the benchmark.

Introduction

Some clients, such as enterprises, cannot share their agent's code, sources or behaviour - with an external optimization process. For them the optimization runs indirectly, without inspecting or directly modifying their agents.

The approach introduces two additional services:

In the blackbox approach, cap-evolve deploys the capabilities to optimize (system prompts, tools, skills) into the Skillberry Store, and the Skillberry Proxy-Agent reads them and injects them into the agent's LLM calls. The benchmark agent is left unchanged; it receives the optimized response from Skillberry.

The optimized capabilities can be used in two ways:

  1. Through the Skillberry service: the client never changes its benchmark. The benchmark is optimized through the responses the Skillberry Proxy-Agent returns.
  2. By exporting the optimized capabilities from the Skillberry Store and patching them back into the client's benchmark.

How it works

The blackbox data flow The agent under test calls the sentinel model with context headers. The Skillberry Proxy-Agent on port 7000 reads the one skill from the Skillberry Store on port 8000, whose tools reach the remote environment service on port 8004, which hosts the benchmark's tool execution. The proxy calls the real model upstream. Dashed arrows are the responses: the environment result back to the tool, the tool result back to the Proxy-Agent, and the optimized response back to the agent. The user simulator calls the gateway directly. agent under test Proxy-Agent :7000 serves my_skill the real model routes requests to the LLM Store :8000 the skill's tools run here remote env service :8004 hosts benchmark tool execution user simulator sentinel + headers upstream call tool call env_<tool>(…) the gateway directly optimized response model reply tool result env result reply request response

The diagram above shows the flow between the benchmark agent and the real model it reaches. The agent receives an optimized response without being modified.

Setup

Assumes cap-evolve is already installed (getting started) and your benchmark's adapter is in .capevolve/project/adapters/.

# 1. Provision the Skillberry Store and Proxy-Agent (clones, venvs, installs)
python -c "import sys; sys.path.insert(0, 'skills/interventions/llm-proxies/blackbox/scripts'); \
import blackbox_env; blackbox_env.provision()"

# 2. Check the stack is ready
python skills/interventions/llm-proxies/blackbox/scripts/run.py

This clones both services under vendor/ and reports, for each one, whether it is provisioned, running and healthy. A run never provisions anything itself: its preflight stops if the stack is not up.

What your benchmark must provide. The blackbox approach checks these, but does not supply them:

  • The agent's LLM calls go to the Proxy-Agent. Other calls (user simulator, judges, verifiers) go straight to the model.
  • The benchmark's environment is reachable over HTTP, because the skill's tools run in the Store, not where the benchmark's state lives.
  • The adapter deploys each candidate in apply(): it makes the candidate the one skill in the Store, then restarts the Proxy-Agent on it.

Configure and run

# What the blackbox approach runs
SPA_MODEL_NAME=<model>              # the model the Proxy-Agent calls on the agent's behalf
SPA_REMOTE_ENV_URL=<url>            # where your benchmark's environment service listens
SKILLBERRY_STORE_PORT=8000          # the Store's port
SKILLBERRY_AGENT_PORT=7000          # the Proxy-Agent's port
# capevolve.yaml — the keys that select the approach
intervention: blackbox
skill_name: <skill>                 # required: the one skill the Proxy-Agent serves

# what to optimize
capabilities: [tools]               # or [system-prompt], [skill-package], ...
capability_path: seed_capability    # the seed; a copy is optimized each iteration
# Run the optimization
cap-evolve run --spec .capevolve/project/capevolve.yaml

Each iteration, cap-evolve deploys the current candidate to the Skillberry Store, and the Proxy-Agent injects it into the agent's LLM calls.
cap-evolve then collects the rewards, gates the edit and repeats. The benchmark itself is never modified.
Output lands in .capevolve/run_<timestamp>/, with report.md, dashboard.html and the optimized candidates.
The run's Config tab shows intervention: blackbox under Delivery.

An onboarding example (τ²-bench)

A coding agent onboards a benchmark for the blackbox approach by following the prompt below and answering blackbox when it asks which approach to use. It installs the benchmark unmodified, seeds the capability from the benchmark's own source, and adds the tailoring the approach needs from outside the benchmark. It then passes the cap-evolve check gate and runs the optimization loop: it proposes edits, keeps only those that clear the significance gate, scores the sealed test once, and writes a report.

Reference: benchmarks/tau2/examples/sierra_airline/PROMPT.md