Skip to content

fix(kubernetes): give the sandbox user a private /sandbox - #5722

Draft
cv wants to merge 1 commit into
NVIDIA:mainfrom
cv:fix/5721-k8s-private-sandbox-home/cv
Draft

cv wants to merge 1 commit into
NVIDIA:mainfrom
cv:fix/5721-k8s-private-sandbox-home/cv

Conversation

@cv

@cv cv commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Draft pending maintainer acceptance of #5721 (state:accepted); the PR Issue Acceptance check is expected to fail until then.

Summary

On Kubernetes the agent container mounted the workspace PVC root at /sandbox, so the provisioner chose the owner and mode of the sandbox user's home. local-path creates the root as root:root 0777, and kubelet doesn't apply fsGroup to its hostPath volumes. New sandboxes now mount a sandbox directory on the PVC, which the non-root workspace-init container creates owned by the sandbox identity with mode 0700, as Docker and Podman create /sandbox.

Related Issue

Closes #5721

Changes

  • openshell-driver-kubernetes: with a workload identity (every sandbox the driver creates), the agent's workspace mount gets subPath: sandbox, and workspace-init runs openshell-sandbox seed-private-workspace. The init container keeps its security context: the sandbox UID and GID, no capabilities, read-only root filesystem. No container in the pod gains root.
  • openshell-sandbox seed-private-workspace (new): if <volume>/sandbox exists, it checks that it is a real directory owned by this UID and stops. Otherwise it removes any leftover <volume>/.sandbox.seeding, creates it with mode 0700, copies the image's /sandbox into it with the existing copy_workspace_tree, fsyncs, and renames it to <volume>/sandbox. The directory exists only after a complete seed, so an interrupted seed starts over on the next Pod start. Unlike seed-workspace, this needs no sentinel and puts no .openshell-initialized file in /sandbox.
  • seed-workspace is unchanged. Image updates and restarts keep a sandbox's stored containers and replace only images, so sandboxes created before this change keep mounting the PVC root and running seed-workspace with the new runtime binary. Their workspaces keep their contents and their current mode; only new sandboxes get the private directory.
  • Conformance: persistent-workspace gains workspace_belongs_to_the_sandbox_user, which checks that /sandbox is owned by id -u and isn't group- or world-writable.
  • Docs (docs/how-it-works/sandboxes/runtimes.mdx): one paragraph stating that contract. Docker, Podman, and Kubernetes create /sandbox with mode 0700. MicroVM gives the image's /sandbox to the sandbox user and keeps the image's mode, so the paragraph says a MicroVM image must not make it group- or world-writable.

Other options I considered: an init container with CAP_CHOWN/CAP_FOWNER running as root could fix the PVC root in place, but the sandbox pod is otherwise entirely non-root, and the Restricted Pod Security Standard forbids it. Relying on fsGroup doesn't work for hostPath-backed PVs. Even where it does, the root becomes root:<gid> with setgid, not owned by the sandbox user.

The PVC is still mounted at two paths (/mnt/openshell-workspace in workspace-init, /sandbox in the agent), so this neither fixes nor worsens #5476. A fix for #5476 that seeds through /sandbox would need to keep the subPath.

Release Notes

On Kubernetes, new sandboxes get a /sandbox owned by the sandbox user with mode 0700 on every StorageClass, as on Docker and Podman. Before, the mode depended on the provisioner; local-path made it root-owned and world-writable. Existing sandboxes keep their workspace as it is.

Testing

  • Checks appropriate to the affected code and behavior pass
    • cargo test -p openshell-sandbox --bin openshell-sandbox workspace: 4 pass, including 3 new tests. They cover a 0777 volume root with a leftover staging directory, which gets a 0700 workspace owned by the caller, the seeded files and symlinks, and no reseed after changes; an image without /sandbox; and a refused symlinked workspace.
    • cargo test -p openshell-driver-kubernetes --lib: 335 pass. The pod-template test now also asserts the subPath, the seed-private-workspace command, and the init container's mount.
    • cargo clippy --all-targets -- -D warnings for both crates and for the conformance crate, and cargo fmt --check: clean.
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable): the conformance test above. I compiled and linted it but didn't run it against a gateway.
  • On a local kind v0.33.0 cluster (kindest/node:v1.37.0, local-path), using this branch's openshell-sandbox binary in pods with the driver's security settings (runAsUser/runAsGroup/fsGroup 10001, OnRootMismatch, no capabilities, read-only root filesystem for the init container):
    • Old layout (PVC root at /sandbox, seed-workspace): /sandbox 777 0:0, with .openshell-initialized inside.
    • New layout (subPath: sandbox, seed-private-workspace): /sandbox 700 10001:10001, with the image's files seeded. The PVC root on the node stays 777 0:0, but the workload can't see it.
    • After a file change and a new Pod on the same PVC, /sandbox was still 700 10001:10001, the change was kept, and nothing was reseeded.
  • Not run: the full gateway Kubernetes E2E and conformance lanes (they need test:e2e-kubernetes), and MicroVM.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

The agent container mounted the workspace PVC root at /sandbox, so the
provisioner chose its owner and mode. local-path creates it root-owned
with mode 0777, and kubelet does not apply fsGroup to its hostPath
volumes, so the sandbox user's home was world-writable without the
sticky bit. Docker and Podman create /sandbox owned by the sandbox user
with mode 0700.

New sandboxes mount the PVC's "sandbox" directory at /sandbox instead.
The non-root workspace-init container creates it with the new
seed-private-workspace subcommand: it seeds a staging directory from the
image as the sandbox identity, sets mode 0700, and renames it into
place, so the directory exists only after a complete seed.
seed-workspace keeps its meaning for sandboxes whose stored Pod template
still mounts the PVC root.

Add a conformance test that /sandbox belongs to the sandbox user and is
not group- or world-writable, and document that contract.

Closes NVIDIA#5721

Signed-off-by: Carlos Villela <[email protected]>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(kubernetes): /sandbox is root-owned with mode 0777 on local-path storage; Docker and Podman create it 0700 for the sandbox user

1 participant