Agent evaluations and benchmarks

Why run each eval task in its own VM?

A separate VM per task means nothing one task writes can affect the next, which keeps results clean. Each sandbox is a full Ubuntu microVM with its own kernel and its own Docker daemon, so harnesses that need Docker run unchanged, and nothing a candidate patch does reaches your machine or other tasks.

Sandboxer's guide on whether its sandboxes fit eval and research teams maps the common jobs to features:

How does a typical eval run work?

A typical run has three steps: build an image once, freeze it as a named snapshot, then fan out. A trimmed version of the guide's flow:

# 1. build and freeze the task image
sbx snapshot <builder-id> eval-task-v1
sbx delete <builder-id> --yes

# 2. one sandbox per task, from the snapshot
id=$(sbx --json new --from eval-task-v1 --type small --ttl 900 | jq -r 'select(.event=="ready") | .id')
sbx exec "$id" "./run_task.sh median"

# 3. copy results out, then delete
sbx scp "$id":out/median.tgz .
sbx delete "$id" --yes

Put a version in the snapshot name, such as task-v1. Reusing a name replaces the snapshot, and a cited result should point to an environment that still exists.

Can you run SWE-bench on Sandboxer?

Yes. Sandboxer's SWE-bench guide runs the official SWE-bench harness inside a sandbox. Because each sandbox has its own Docker daemon, the harness pulls the per-task image, applies the candidate patch, runs the tests and grades FAIL_TO_PASS and PASS_TO_PASS as designed.

The guide uses a large sandbox (8 vCPU, 16 GB, 125 GB disk), because task images are a few gigabytes each and the harness runs several containers. To scale out, save one prepared sandbox as a snapshot and start one VM per task. The guide is clear that its timings are single runs, not a benchmark.

How do you run best-of-N or RL episodes?

Use forks to branch from one state. For best-of-N attempts, prepare the repo once, fork it once per attempt, run the tests in each fork, save each diff and score, and keep the best. For RL and eval environments, each episode runs your policy and grader in a fresh sandbox, writes a reward file you read back, and is deleted.

A fork copies the disk, not the memory: it boots fresh, as Sandboxer's guide on forking a running sandbox explains.

How fast can you fan out?

Your plan limits how many sandboxes run at once and how many you can start per minute, hour and day. On the $500 plan, for example, 1,000 can run at once with 65 starts a minute. Check what is left before a burst:

sbx limits --json | jq '.starts'

Over a limit, the API returns 429 with a Retry-After header. Sandboxer's guide on rate limits and concurrency caps suggests sizing a client-side semaphore from GET /limits. Full plan figures are on pricing.

How do you keep an eval sweep from running up a bill?

Give every task sandbox a short deadline, such as --ttl 900 for 15 minutes, so a hung task stops on its own, and delete each sandbox once its results are copied out. Check sbx limits before a big sweep: when the balance reaches zero with auto-refill off, new creates, resumes and forks are refused with 402 billing_required, and running sandboxes are stopped after a 24-hour grace period with memory and disk kept. In an organization, an owner can also cap each member's machine time. See how to stop an AI agent from overspending and Sandboxer's billing docs.

What should you record for reproducible results?

• The snapshot name and version the task ran on
• CPU model, kernel and dependency hashes, since CPU types can differ between machines
• Whether a sandbox was cold-booted, using memoryRestored and resumeReason; the guide advises not stopping a VM mid-task

There are no GPUs on Sandboxer, so evaluations that need local GPU inference belong elsewhere. Billing is per second, CPU only.