Skip to content

feat(helm): size cluster sandbox capacity from Helm values - #5756

Open
FrostGod wants to merge 7 commits into
mainfrom
feat/5748-configurable-supervisor-pool-limits/FrostGod
Open

FrostGod wants to merge 7 commits into
mainfrom
feat/5748-configurable-supervisor-pool-limits/FrostGod

Conversation

@FrostGod

@FrostGod FrostGod commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Make shared supervisor pool capacity configurable from Helm, so operators can size a cluster by gateways, supervisors, and sandboxes per supervisor.

Related Issue

Closes #5748

Changes

  • New Helm values under supervisor.pool:
    • maxInstancesPerMember: sandboxes per supervisor, 1–1024, default 128
    • maxMembersPerGateway: supervisors per gateway, 1 or more, default 32
  • Total capacity is live gateway replicas × supervisors per gateway × sandboxes per supervisor, never more than 131072.
  • New sandboxes go to the least-loaded supervisor and never past its limit.
  • At capacity, sandbox create fails right away with FAILED_PRECONDITION: sandbox capacity reached; add supervisors or gateway replicas, or raise the supervisor pool limits. The gateway log says which limit was hit.
  • Message buffers are sized from the limits.
  • Chart defaults:
    • supervisor.replicas: 2
    • supervisor autoscaling off, maxReplicas: 4
    • supervisor resources: requests 250m / 768Mi, memory limit 1.5Gi
  • Rendering fails when supervisor replicas (or autoscaling maxReplicas) exceed gateway replicas × maxMembersPerGateway.
  • Added a values-scale-10k.yaml profile: 2 gateways × up to 40 supervisors each × 128 sandboxes = 10,240.
  • Docs updated: gateway configuration, the HA guide, and the cluster and dev-environment skills.

Release Notes

Helm users can set shared supervisor capacity with supervisor.pool.maxInstancesPerMember and supervisor.pool.maxMembersPerGateway. Creates beyond capacity fail with "sandbox capacity reached".

Testing

  • Rust unit tests for openshell-supervisor-control, the shared supervisor host, server config, and placement; clippy clean
  • helm unittest (503 tests), mise run helm:lint, test-gateway-config-parser.sh, test-supervisor-pool-scaling.sh
  • Local k3d cluster with 10–20 sandboxes across several limit settings: even spread (7/6/7 on 3 supervisors), creates refused at the limit
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…imits

Add [openshell.gateway.supervisor_pool] max_instances_per_member (16-1024,
default 128) and max_members_per_gateway (1-64, default 32). The pool cap is
live gateway replicas * members * instances, never above 131072. Placement
picks the least-loaded member with room and never charges a member past its
cap. Supervisors take the same cap through --pool-max-instances and size
their gateway stream buffers from it.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
Render supervisor.pool.maxInstancesPerMember and maxMembersPerGateway into the
gateway config and supervisor env, validate their ranges, and fail the render
when supervisor replicas or HPA maxReplicas exceed gateway replicas *
maxMembersPerGateway. Default to two supervisors with configurable resources
(250m/768Mi requests, 1.5Gi memory limit) and add values-scale-100k.yaml.

Closes #5748

Signed-off-by: divesh <dgude@nvidia.com>
Lower max_instances_per_member and supervisor.pool.maxInstancesPerMember to
1..=1024. Gateway message and status-report bounds keep the floor a cap of 16
gave, since snapshots also carry the pool-wide route directory. Document how
to add gateway replicas for more supervisors.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…ool limits plumbing

Only sandboxes per supervisor stays bounded (1-1024). Members per gateway
accepts any positive value; the 131072 pool ceiling still applies.
Placement refusals now say "sandbox capacity reached" and name the
limit that was hit.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…imits

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

@FrostGod
FrostGod requested a review from drew November 19, 2026 06:51
Two gateways with up to 40 supervisors each, at 128 sandboxes per
supervisor, hold 10,240 sandboxes.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

Copy link
Copy Markdown
Collaborator Author

/ok to test c17f546

…tion

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(helm): size cluster sandbox capacity from Helm values

1 participant