Repository navigation
feat(gateway): broker stable sandbox SPIFFE provider tokens - #4732
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4732.docs.buildwithfern.com/openshell |
db063a9 to
8dfbd78
Compare
a876a1a to
437619a
Compare
3da8000 to
5fbece5
Compare
5fbece5 to
1899b5b
Compare
96ba1b7 to
e0b6cec
Compare
b4c1940 to
b733887
Compare
b733887 to
8e6786b
Compare
8e6786b to
6dd2cbb
Compare
6dd2cbb to
85cba03
Compare
a78134c to
03c6d6e
Compare
Preserve the SPIFFE RPC alongside service discovery and update the combined public schema inventory without changing durable fingerprints. Signed-off-by: Taylor Mutch <[email protected]>
EmilienM
left a comment
There was a problem hiding this comment.
Another iteration needed: 4 major, 19 minor (inline, 1 below)
The Broker API rework removed the hostPath and node-coupling problems from round 1, and the cache-binding fixes hold up. The main problem now is that the Broker connection pins one SPIRE agent's ID, so the documented Service setup only works on a single-node cluster. The other majors are per-request costs on the shared-supervisor path (catalog scans, the global gRPC rate limiter) and the wire format of the new RPC's credential_key.
Outside the diff:
- Minor:
docs/how-it-works/providers/profiles.mdx:705, still says token grants need the sandbox supervisor's own Workload API socket, as dodocs/kubernetes/access-control.mdx:37anddeploy/helm/openshell/README.md.gotmpl:535; a short note linking to the new Broker section would cover shared supervisors.
How this was reviewed
- Commit
d7b0c95, round 2: a full pass, since the branch was rebased after round 1's9b60d8d - Deep review, because it touches
AGENTS.md,.agents/skills/helm-dev-environment/SKILL.mdandcrates/openshell-core/src/dynamic_credential_key.rs - Passes: architecture, performance, security, scalability, rust, kubernetes and supply-chain, then correctness, then an advanced defensive review
- Independent skeptics checked each candidate finding (3 with a reproducer for blocking and major ones, 2 for architecture and scalability ones, 1 for minor ones); 19 of 23 held up
- Repo rules read from
main: AGENTS.md, CLAUDE.md, CONTRIBUTING.md, docs/CONTRIBUTING.mdx - Not covered: nothing skipped
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
…n errors Signed-off-by: Taylor Mutch <[email protected]>
… grants Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
… teardown Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
…ation Signed-off-by: Taylor Mutch <[email protected]>
|
Pushed the review fixes and node-local Broker changes through 14a87e4. Validated the node-local path against a four-node GKE cluster. The targeted SPIFFE test binary passed 8/8 tests (one live integration scenario and seven shared-helper tests), with no failures or skips. The live scenario covered:
The remote SIGKILL fixture uses an unprivileged sidecar sharing the supervisor Pod's process namespace and UID; it requires no host PID namespace, runtime socket, or additional capabilities. Additional local checks passed:
The live GKE run used IPv4. IPv6 endpoint handling is covered by unit tests; OpenShift The PR description now shows the application flow and explains that gateway-to-SPIRE traffic stays node-local. The GKE cluster and deployed fixtures have been left in place. |
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
TaylorMutch
left a comment
There was a problem hiding this comment.
Approved at be94c73. Branch Checks, Helm Lint, E2E, and the GPU gate are green. Review threads are resolved, including the acquisition-failure fan-out and lazy-channel fixes. The small Kubernetes runtime-format duplication remains an accepted scope decision.
I accept the documented Rust generated-server-trait and TypeScript injected-client source-compatibility impacts of adding GetSandboxProviderAccessToken to OpenShell. The experimental SPIRE Broker prerequisite and outstanding live OpenShift validation remain documented.
Multi-node GKE validation: #4732 (comment)
…ecrets/mrunalp Resolve the schema inventory conflict with #4732 by combining the RPC counts and recomputing the reviewed public fingerprint. Adapt to #5668, which added Basic client authentication to OAuth refresh: count a client_secret supplied by a workspace secret as present when validating token_endpoint_auth_method, and test a Basic mint whose client secret comes from a workspace secret. Update the new request_token test and refresh request literal for the merged signatures. Signed-off-by: Mrunal Patel <[email protected]>
Summary
Enable shared-supervisor Kubernetes sandboxes to obtain provider access tokens using stable sandbox SPIFFE identities. The gateway obtains sandbox JWT-SVIDs through SPIRE's experimental SPIFFE Broker API and performs provider-token exchanges. Supervisors authenticate with each sandbox generation's credentials and receive only final provider access tokens; workloads receive no SPIFFE socket or credentials.
Related Issue
Closes #4731
Changes
GetSandboxProviderAccessTokento the OpenShell service. Validate the signed session against persisted runtime generation, credential epoch, token lineage, and active state. Resolve provider attachment, credential key, endpoint, audience, and scopes from gateway configuration.<prefix>/gateway/<gateway-id>/sandbox/<sandbox-id>and request identity using the Sandbox resource's persisted Kubernetes UID. Callers cannot supply workload references or selectors.openshell-crypto, then verify returned JWT-SVID signatures, audience, expiry, and exact sandbox subject.openshell-spiffeand the Broker adapter intoopenshell-spire. Update published configuration docs, Helm guidance, troubleshooting, and integration fixtures.Application flow
This shows a cache miss. Gateway-to-SPIRE traffic stays on the gateway's node: the Workload API uses its CSI-mounted Unix socket, and the Broker API uses mTLS directly to the node address supplied by Kubernetes' Downward API. No Kubernetes Broker Service is used in node-local mode. Workloads and shared supervisors can run on other nodes.
sequenceDiagram participant W as Sandbox workload participant S as Shared supervisor participant G as Gateway participant A as Gateway node's SPIRE agent participant K as Kubernetes API participant T as Provider token service participant U as Upstream API W->>S: Make an HTTP request S->>G: Request token using sandbox session JWT G->>G: Authorize session, provider and configured grant G->>A: Local Workload API: obtain gateway identity and trust bundles G->>A: Node-local Broker over mTLS: request sandbox identity by CR UID A->>A: Check allowlisted gateway SPIFFE identity A->>K: Attest Sandbox CR and check gateway authorization K-->>A: Attestation and authorization results A-->>G: Sandbox JWT-SVID and verification bundles G->>G: Verify sandbox identity, signature, audience and expiry G->>T: Perform configured OAuth2 grant or token exchanges T-->>G: Final provider access token G->>G: Recheck configuration and session authorization G-->>S: Final access token and remaining lifetime S->>U: Forward HTTP request with injected token U-->>S: Response S-->>W: ResponseThe gateway performs the RFC 7523 grant or both RFC 8693 exchange stages. Shared supervisors receive only the final provider token; sandbox workloads and shared supervisors receive no SPIFFE credentials or Workload API mount.
Configuration and scope
The feature is disabled unless
[openshell.gateway.spiffe_broker]is configured; there is no compile-time feature flag. Gateway session JWTs with a positive TTL are required.Operators must enable SPIRE's experimental Broker endpoint, authorize the gateway identity, configure Kubernetes attestation with
access_policy = "enforced", and provision registrations for sandbox identities. The agent needs Sandbox lookup and SubjectAccessReview permissions; the gateway identity needs narrowly scopedimpersonate-via-spireauthorization. Authorize the Broker peer with an exact SPIFFE ID or a cluster-scoped SPIFFE path prefix. Configurenode_local_portto reach the gateway node’s agent directly, withOPENSHELL_NODE_IPinjected fromstatus.hostIP. This path uses no Broker Service. An explicit HTTPSendpointremains available as an alternative. Use a node alias to distribute sandbox registrations to every node where a gateway replica may run.The gateway uses a CSI Workload API mount. Shared supervisors and sandbox workloads require no SPIFFE mounts. RFC 8693 profiles use the same configured
spiffe_broker.workload_api_socket_pathfor the gateway's component JWT-SVID.This change supports Kubernetes shared supervision. It does not automatically provision SPIRE registrations or enable other compute drivers. Already-issued provider bearer tokens retain their normal expiration. Kubernetes Restricted Pod Security admission was validated; OpenShift
restricted-v2SCC compatibility has not been validated.The public RPC adds a method to generated Rust server traits and the full TypeScript client interface; downstream handwritten implementations and mocks may require updates.
Connection pooling, broader cache improvements, generated upstream Broker bindings, and inclusion of the SPIRE fixture in default CI remain follow-ups.
Testing
Local validation of the node-local changes at
14a87e4eb:spiffe-poccluster. Coverage includes gateway relocation and fresh acquisition on every node without a Broker Service, node-local Broker egress restrictions, local-agent replacement, abrupt supervisor failure, credential rotation, and stale-session fencing.Validation of the implementation at
17c8b7b2c:The subsequent commit
4532524dconly corrects troubleshooting advice and a test description;git diff --checkpassed.