CI and testing

Why run tests in a sandbox?

A sandbox gives tests a clean, full machine that can do things hosted runners often cannot, and it disappears afterwards. Each one has Docker with Compose, root access, systemd and nested KVM, plus Chrome, Node, Python, Go and Rust preinstalled. Pull-request code runs in a VM with none of your secrets if you create it with --no-env.

Sandboxer's guide for QA and test automation teams adds a few habits: copy JUnit files, screenshots and traces out before deleting; rerun suspect tests in a clean VM to measure flakiness; and keep a failing VM by stopping it instead of deleting it, so you can inspect it later for free.

How do you use Sandboxer from GitHub Actions?

Sandboxer's GitHub Actions guide describes two patterns:

Start with a key that uses the ci preset, which can create sandboxes and delete only the ones it created:

sbx api-key create github-actions --preset ci --ttl 90d

What does the hosted-runner pattern look like?

The job installs sbx, creates a sandbox, copies the checkout in, runs the tests and deletes the sandbox even on failure. The core steps:

id="$(sbx new --type default --ttl 1800 --no-env --json | jq -r 'select(.event=="ready") | .id')"
tar -czf - . | sbx ssh "$id" -- "mkdir -p repo && tar -xzf - -C repo"
sbx exec "$id" --cwd repo --timeout 600 "npm ci && npm test"
sbx delete "$id" --yes

Because sbx exec exits with the remote command's code, a failing test fails the step. The --ttl 1800 backstop stops the sandbox after 30 minutes if the job dies before cleanup. The guide also recommends setting the step shell to bash so a failed install download fails the step.

How do you shard a slow test suite?

Install the project once, save it as a named snapshot, start one sandbox per shard, and merge the reports. Sandboxer's sharding guide uses pytest-split:

sbx snapshot <sandbox-id> tests-ready
id=$(sbx new --from tests-ready --type small --json | jq -r 'select(.event=="ready") | .id')
sbx exec "$id" "pytest --splits 2 --group 1 --junitxml=junit-1.xml"

In that guide's run, two small shards cut the networkx 3.5 suite from 96.9 seconds to 59.3 seconds. Each start counts against your plan's per-minute start limit, so pace large fan-outs.

Can every pull request get a preview URL?

Yes. CI creates one small sandbox per pull request, labeled with the PR number, builds and serves the app, then runs sbx host to get an HTTPS URL to post on the PR. Idle previews pause and wake on the next visit. When the PR closes, CI finds the sandbox by label and deletes it. See preview environments.

sbx host <sandbox-id> 3000

The URL takes the form https://<subdomain>-<port>.box.sandboxer.com. It is private by default and needs its token, or you can add --public.

What do CI runs cost?

Per second of runtime at the size's hourly rate: $0.018 for small, $0.036 for default, $0.072 for large. A large sandbox running a 3-hour build and then stopped for a month uses 6 hours of default time, which Sandboxer's billing docs price at $0.22. See pricing.