Storage, health, and revisions#
Three features make a sandbox something you can run for weeks rather than minutes: a disk that survives restarts, a probe that tells you whether the agent is actually working, and a revision history you can roll back.
Persistent volumes#
A sandbox's container filesystem is discarded when it stops. A persistent volume is not.
volumes:
- mountPath: /data
size: 5Gi
size is optional. Without it the volume gets the install's default, sandbox.k8s.agentVolumeSize (10Gi unless set). A size above the install's ceiling, sandbox.k8s.disk, is rejected with 400. While sandbox.k8s.disk is empty, the ceiling is the default size, so a larger volume needs the ceiling raised. See Sandbox runtime.
Or with a flag, or the Volume size field of the create form in the admin UI:
Create a sandbox in production from ghcr.io/acme/agent:1.2 running
"./start.sh" with 500m CPU and 2Gi memory, and give it a 5Gi
persistent volume mounted at /data.
nexusctl sandbox create --project production \
--image ghcr.io/acme/agent:1.2 \
--command "./start.sh" \
--cpu 500m --memory 2Gi \
--volume /data:5Gi
The path must be absolute. A sandbox has at most one volume, and its path is set at create. Local Docker ignores the size.
An update can grow the volume in place, without a restart: send the same mountPath with a larger size, up to sandbox.k8s.disk. The StorageClass must set allowVolumeExpansion; if it does not, the update gets 400 with the reason from Kubernetes and none of it applies. A volume cannot shrink or be added or removed after create, and growing it and changing credentials take two updates. The new size can take a few minutes to show inside the sandbox.
curl -fsS -X PATCH \
-H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
-H "Content-Type: application/json" \
https://agents.example.com/v1/projects/$PROJECT_ID/sandboxes/$SANDBOX_ID \
-d '{"volumes": [{"mountPath": "/data", "size": "20Gi"}]}'
cat > grow.yaml <<'EOF'
volumes:
- mountPath: /data
size: 20Gi
EOF
nexusctl sandbox update nightly-refactor --project production -f grow.yaml
The admin UI's edit dialog has a Volume size field.
What the volume gets you:
- Survives stop and start. Stop a sandbox to release compute, start it later, and the agent resumes on the same disk.
- Survives image upgrades. A new revision reuses the volume, so an agent's memory, checkpoints, and working files outlive the version that wrote them.
- Deleted with the sandbox. The volume is part of the sandbox's lifecycle, not separate from it.
Put anything the agent must not lose under the mount path — its memory store, its task state, its workspace. Anything outside it is scratch.
The storage class comes from sandbox.k8s.agentVolumeStorageClass, falling back to the cluster default. See Install on Kubernetes.
Health checks#
Without a probe, "running" means the process has not exited. That is a weak signal for an agent that has hung, wedged on a lock, or crashed its HTTP server while its supervisor stays up.
healthCheck:
type: http
http:
path: /healthz
port: 8080
--health-check path=/healthz,port=8080,interval=10,initial-delay=30,timeout=5,failures=3
Pass --health-check none to leave probing off.
| Field | Default | Range |
|---|---|---|
type |
none |
http or none |
intervalSeconds |
10 | 1–3600 |
initialDelaySeconds |
30 | 0–3600 |
timeoutSeconds |
5 | 1–300 |
failureThreshold |
3 | 1–100 |
Give initialDelaySeconds enough room for the agent to start. An agent that pulls models or warms a cache on boot will fail a probe that starts checking too early, and get restarted before it ever becomes ready.
Reading health#
nexusctl sandbox describe surfaces the latest health snapshot the pod watcher pushed:
| Field | Tells you |
|---|---|
agentRunning |
Whether the agent process is up. |
startedAt / lastActivityAt |
When it started, and when it last did something. |
exitStatus |
Exit code and signal, when it has exited. |
probe.status |
Result of the most recent HTTP probe. |
probe.restartCount |
How many times the container has restarted. |
probe.lastError |
Why the last probe failed. |
imageDigest |
The resolved digest actually running — the answer to "is this really the image I pushed?". |
errorReason |
Provisioner error for a sandbox in error. |
Revisions#
Every change to a sandbox's image, command, resources, environment, health check, or exposed ports creates a numbered revision and restarts the sandbox on it. Label-only and policy-only changes are metadata and do not restart anything.
curl -fsS -H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
https://agents.example.com/v1/projects/$PROJECT_ID/sandboxes/$SANDBOX_ID/revisions
List the revisions of the nightly-refactor sandbox in production
and tell me what changed between the last two.
Each revision is a full snapshot of the configuration that was live, so the history answers "what was this agent running last Tuesday?" without a separate change log.
Rollback#
curl -fsS -X POST \
-H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
-H "Content-Type: application/json" \
https://agents.example.com/v1/projects/$PROJECT_ID/sandboxes/$SANDBOX_ID/rollback \
-d '{"revisionNum": 3}'
Roll the nightly-refactor sandbox in production back to revision 3.
Rollback restores the image, command, resources, environment, health check, and exposed ports of that revision together. The persistent volume is not rolled back — the disk stays as the newer version left it.
Revisions and rollback are API and MCP surfaces
nexusctl has no revisions or rollback verb today. Use the REST endpoints above, or the list_sandbox_revisions and rollback_sandbox tools over MCP.
Spend#
An agent that runs for weeks spends money for weeks. Cap it per sandbox:
curl -fsS -X PUT \
-H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
-H "Content-Type: application/json" \
https://agents.example.com/v1/orgs/$ORG_ID/spending-limits \
-d '{"actorType": "sandbox", "actorId": "'"$SANDBOX_ID"'",
"period": "month", "limitCents": 5000}'
Cap the nightly-refactor sandbox in production at $50 a month.
nexusctl sandbox set-limit nightly-refactor \
--project production \
--period month --limit-cents 5000
On breach, the inference proxy returns 429 and the sandbox keeps running — it loses model access, not its process and not its disk.
Check what it has spent:
curl -fsS -G -H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
https://agents.example.com/v1/orgs/$ORG_ID/usage-costs \
-d actorType=sandbox -d actorId=$SANDBOX_ID
How much has the nightly-refactor sandbox spent this month?
nexusctl sandbox spend nightly-refactor --project production
See Spending limits for org, user, and agent scopes.
Image updates#
A sandbox can follow the tag of its image, such as ghcr.io/acme/agent:latest, and the platform checks the registry for a new digest behind it. It is off by default. The image must have a tag and no digest of its own.
imageUpdate |
Effect |
|---|---|
null (default) |
Off. The runtime pulls the image as written, so a start can run a newer digest. |
{auto: false} |
A new digest is shown in imageUpdateStatus.available. Take it with Update now in the admin UI, the update_sandbox_image MCP tool, or the request below. |
{auto: true} |
The platform also rolls the new digest out as a new revision. |
curl -fsS -X POST \
-H "Authorization: Bearer $LENS_AGENTS_TOKEN" \
-H "Content-Type: application/json" \
https://agents.example.com/v1/projects/$PROJECT_ID/sandboxes/$SANDBOX_ID/update-image \
-d '{"digest": "<imageUpdateStatus.available>"}'
The digest must be the one in imageUpdateStatus.available; any other is refused with 409, and so is a request while another change to the sandbox is still rolling out. On a stopped sandbox it applies at the next start.
A tag that an auto sandbox follows is checked every hour, another every six hours. Docker Hub is asked at most every 20 minutes.
An automatic rollout reaches only a started sandbox. It waits until the sandbox has been idle for 10 minutes, but no longer than an hour after the digest was found, and restarts it on a revision that pins the digest (repo:tag@sha256:...). The sandbox's image stays as written. A rollout is a restart: only the volume is kept. The platform never moves a sandbox to an older digest, and starts at most 10 rollouts a minute.
If the sandbox goes to error within 10 minutes of a rollout, or has not started by then, the platform goes back to the previous revision and holds that digest in imageUpdateStatus.heldDigest, so it is not rolled out again. Changing image, or taking the held digest with Try again in the admin UI or the request above, clears the hold. A rollback turns auto off. Every automatic rollout and return is audited as sandbox.auto_update.
Changing imageUpdate does not restart the sandbox.
Related#
- Sandbox spec — declaring volumes and probes in a file
- Exposed ports — the port a health check usually probes
- Spending limits — budgets by org, user, agent, and sandbox
- Install on Kubernetes — storage classes and sandbox ceilings