Harbor integration

Use Harbor as cap-evolve's sandboxed evaluation backend — any agent, any benchmark, any environment (-e docker, -e openshift, -e daytona, -e modal).

How it works

cap-evolve optimizes; Harbor evaluates. The Harbor adapter (templates/adapters/harbor/adapter.py) is the bridge — its run_batch() calls harbor run as a subprocess each time cap-evolve needs an evaluation. You never call harbor run directly; cap-evolve does it for you.

you run:   cap-evolve run --spec .capevolve/project/capevolve.yaml
             │
cap-evolve:  hill_climb_loop()
             ├── evaluate → adapter.run_batch()
             │     └─ harbor run -d <dataset> -a <agent> -m <model> -e <env>
             │           ├─ container/pod 1: agent + verifier → reward
             │           ├─ container/pod 2: agent + verifier → reward
             │           └─ ...
             │     adapter reads reward.json → cap-evolve Score
             ├── gate: accept if mean(Δ) > k × SE(Δ)
             ├── optimizer edits SKILL.md based on failed trajectories
             └── repeat → finalize on sealed test split

Setup

Assumes cap-evolve is already installed (getting started).

# 1. Install Harbor + adapter utilities
uv tool install harbor              # CLI (v0.20+)
pip install ./capevolve_harbor

# 2. Copy the Harbor adapter into your project
cp -r templates/adapters/harbor/* .capevolve/project/adapters/

This places adapter.py (the Harbor bridge) and a seed SKILL.md into your project. The adapter reads all Harbor config from environment variables at runtime.

Configure and run

# What Harbor runs
export HARBOR_DATASET=<org/dataset>        # registry: swe-bench/swe-bench-verified
                                            # or local: /path/to/dataset
export HARBOR_AGENT=<agent>                 # claude-code, codex, opencode, ...
export HARBOR_MODEL=<model>                 # claude-sonnet-4-6, gpt-4.1, ...
export HARBOR_TASK_IDS=<task1,task2,...>     # which tasks to optimize on
export HARBOR_TASK_PREFIX=<dataset-org>     # e.g. swe-bench

# Where Harbor runs
export HARBOR_EXTRA_FLAGS="-e docker -n 4"  # local Docker
# or: "-e openshift -n 5 --ae KEY=VALUE"   # OpenShift pods
# or: "-e daytona -n 100"                  # cloud

# Run the optimization
cap-evolve run --spec .capevolve/project/capevolve.yaml

Each iteration, cap-evolve calls harbor run with the current candidate skill, collects rewards, gates the edit, and repeats. Output lands in .capevolve/run_<timestamp>/ with report.md, dashboard.html, and the optimized candidates/best/SKILL.md.

Running on OpenShift

Use -e openshift to run task pods on your cluster. The orchestrator (cap-evolve + Harbor CLI) runs as a Job; Harbor creates one task pod per benchmark trial on the same cluster.

RBAC (once per namespace)

oc apply -f openshift/manifests/service-account.yaml          # harbor-task SA + SCC
oc apply -f openshift/manifests/harbor-orchestrator-sa.yaml   # orchestrator SA + RBAC
oc adm policy add-scc-to-user harbor-task-scc -z harbor-task
oc adm policy add-scc-to-user harbor-task-scc -z harbor-orchestrator

Orchestrator image

Adds Harbor + oc to the cap-evolve runner image:

FROM quay.io/<your-org>/cap-evolve-runner:latest
USER root
RUN apt-get update -qq && apt-get install -y -qq curl && \
    curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz \
      | tar xzf - -C /usr/local/bin oc kubectl && \
    apt-get remove -y curl && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir "harbor>=0.20.0"
COPY capevolve_harbor/ /app/capevolve_harbor/
COPY templates/adapters/harbor/ /app/templates/adapters/harbor/

podman build --platform linux/amd64 -t quay.io/<your-org>/cap-evolve-harbor-runner:latest .
podman push quay.io/<your-org>/cap-evolve-harbor-runner:latest

Credential patterns (--ae flags)

For on-cluster model serving, see vLLM on OpenShift.

Configuration

Benchmarks and agents

Any Harbor dataset and any Harbor agent work — set HARBOR_DATASET and HARBOR_AGENT.

References

• Harbor documentation
• Harbor agents
• Harbor datasets
• Harbor adapter source
• OpenShift guide — vLLM, RBAC, credentials, Job manifest
• agent-eval-harness Harbor workflow
• coding_agent_bench — SWE-bench on OpenShift