Repository navigation
Conversation
The agent container mounted the workspace PVC root at /sandbox, so the provisioner chose its owner and mode. local-path creates it root-owned with mode 0777, and kubelet does not apply fsGroup to its hostPath volumes, so the sandbox user's home was world-writable without the sticky bit. Docker and Podman create /sandbox owned by the sandbox user with mode 0700. New sandboxes mount the PVC's "sandbox" directory at /sandbox instead. The non-root workspace-init container creates it with the new seed-private-workspace subcommand: it seeds a staging directory from the image as the sandbox identity, sets mode 0700, and renames it into place, so the directory exists only after a complete seed. seed-workspace keeps its meaning for sandboxes whose stored Pod template still mounts the PVC root. Add a conformance test that /sandbox belongs to the sandbox user and is not group- or world-writable, and document that contract. Closes NVIDIA#5721 Signed-off-by: Carlos Villela <[email protected]>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Kubernetes the agent container mounted the workspace PVC root at
/sandbox, so the provisioner chose the owner and mode of the sandbox user's home. local-path creates the root asroot:root 0777, and kubelet doesn't applyfsGroupto its hostPath volumes. New sandboxes now mount asandboxdirectory on the PVC, which the non-rootworkspace-initcontainer creates owned by the sandbox identity with mode0700, as Docker and Podman create/sandbox.Related Issue
Closes #5721
Changes
openshell-driver-kubernetes: with a workload identity (every sandbox the driver creates), the agent's workspace mount getssubPath: sandbox, andworkspace-initrunsopenshell-sandbox seed-private-workspace. The init container keeps its security context: the sandbox UID and GID, no capabilities, read-only root filesystem. No container in the pod gains root.openshell-sandbox seed-private-workspace(new): if<volume>/sandboxexists, it checks that it is a real directory owned by this UID and stops. Otherwise it removes any leftover<volume>/.sandbox.seeding, creates it with mode0700, copies the image's/sandboxinto it with the existingcopy_workspace_tree, fsyncs, and renames it to<volume>/sandbox. The directory exists only after a complete seed, so an interrupted seed starts over on the next Pod start. Unlikeseed-workspace, this needs no sentinel and puts no.openshell-initializedfile in/sandbox.seed-workspaceis unchanged. Image updates and restarts keep a sandbox's stored containers and replace only images, so sandboxes created before this change keep mounting the PVC root and runningseed-workspacewith the new runtime binary. Their workspaces keep their contents and their current mode; only new sandboxes get the private directory.persistent-workspacegainsworkspace_belongs_to_the_sandbox_user, which checks that/sandboxis owned byid -uand isn't group- or world-writable.docs/how-it-works/sandboxes/runtimes.mdx): one paragraph stating that contract. Docker, Podman, and Kubernetes create/sandboxwith mode0700. MicroVM gives the image's/sandboxto the sandbox user and keeps the image's mode, so the paragraph says a MicroVM image must not make it group- or world-writable.Other options I considered: an init container with
CAP_CHOWN/CAP_FOWNERrunning as root could fix the PVC root in place, but the sandbox pod is otherwise entirely non-root, and the Restricted Pod Security Standard forbids it. Relying onfsGroupdoesn't work for hostPath-backed PVs. Even where it does, the root becomesroot:<gid>with setgid, not owned by the sandbox user.The PVC is still mounted at two paths (
/mnt/openshell-workspaceinworkspace-init,/sandboxin the agent), so this neither fixes nor worsens #5476. A fix for #5476 that seeds through/sandboxwould need to keep thesubPath.Release Notes
On Kubernetes, new sandboxes get a
/sandboxowned by the sandbox user with mode0700on every StorageClass, as on Docker and Podman. Before, the mode depended on the provisioner; local-path made it root-owned and world-writable. Existing sandboxes keep their workspace as it is.Testing
cargo test -p openshell-sandbox --bin openshell-sandbox workspace: 4 pass, including 3 new tests. They cover a0777volume root with a leftover staging directory, which gets a0700workspace owned by the caller, the seeded files and symlinks, and no reseed after changes; an image without/sandbox; and a refused symlinked workspace.cargo test -p openshell-driver-kubernetes --lib: 335 pass. The pod-template test now also asserts thesubPath, theseed-private-workspacecommand, and the init container's mount.cargo clippy --all-targets -- -D warningsfor both crates and for the conformance crate, andcargo fmt --check: clean.kindest/node:v1.37.0, local-path), using this branch'sopenshell-sandboxbinary in pods with the driver's security settings (runAsUser/runAsGroup/fsGroup10001,OnRootMismatch, no capabilities, read-only root filesystem for the init container):/sandbox,seed-workspace):/sandbox 777 0:0, with.openshell-initializedinside.subPath: sandbox,seed-private-workspace):/sandbox 700 10001:10001, with the image's files seeded. The PVC root on the node stays777 0:0, but the workload can't see it./sandboxwas still700 10001:10001, the change was kept, and nothing was reseeded.test:e2e-kubernetes), and MicroVM.Checklist