Skip to content

feat: retry sandbox creates on mutation lock timeouts - #5745

Open
EmilienM wants to merge 5 commits into
NVIDIA:mainfrom
EmilienM:feat/5697-create-retry/EmilienM
Open

EmilienM wants to merge 5 commits into
NVIDIA:mainfrom
EmilienM:feat/5697-create-retry/EmilienM

Conversation

@EmilienM

Copy link
Copy Markdown
Contributor

Summary

sandbox create and the Rust SDK's create methods now retry a create that loses the gateway mutation lock, so burst creation under contention no longer needs a retry loop in every caller. The gateway keeps the machine-readable retry signal when failed-create cleanup rewrites the error, and both lock-timeout producers carry it.

Related Issue

Closes #5697

Changes

  • Server: both SSH identity lock acquisitions map LockTimeout to UNAVAILABLE + MUTATION_LOCK_TIMEOUT + RetryInfo, and compensate_failed_create preserves ErrorInfo/RetryInfo on every rebuilt status
  • Core: the reason constant, an exact-match predicate on tonic::Status, a detail-preserving rebuild helper, and the retry policy itself (delay floor with jitter, backoff, 5-attempt cap, wall-clock budget, AlreadyExists phase decisions), shared by the SDK and the CLI
  • SDK: opt-in create_sandbox_with_retry / create_sandbox_from_template_with_retry on both clients, CreateRetryOptions with a 60s default budget, and SdkError::mutation_lock_timeout(); retryable() keeps its auth-only contract
  • CLI: retry on by default, controlled by --create-retry-timeout, --no-create-retry, and OPENSHELL_CREATE_RETRY_TIMEOUT (60s default, 24h clamp); progress on stderr only; a named create waits out a holder that is deleting or still provisioning; a --from create re-stages its archive when the gateway reports the staging token spent
  • Docs: SDK error contract, HA page, sandbox overview, CLI skill, debug skill, SDK README

Release Notes

sandbox create now retries for up to 60 seconds when the gateway mutation lock is busy and waits out a same-name sandbox that is still being deleted. The Rust SDK gains opt-in retrying create methods. MUTATION_LOCK_TIMEOUT failures keep their reason and retry delay even after failed-create cleanup rewrites the message.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Tests cover the reason predicate, detail survival through every compensation branch, retry timing (floor, jitter, backoff, attempt cap), budget exhaustion, the AlreadyExists phase matrix (deleting, provisioning, missing, live, different-id holder), staging-token re-staging, and deadline-bounded holder polls. The PostgreSQL lock suite passed 48/48. The one local failure is a pre-existing z3 solver timeout in openshell-prover, reproduced on clean origin/main. No e2e lane: this is client retry policy plus server error mapping, covered by the SDK/CLI mock suites and the server compensation tests.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

…sion layer

Add the shared primitives that let clients retry sandbox creates that
lose the gateway mutation lock budget. rpc_error gains the
MUTATION_LOCK_TIMEOUT reason constant, an exact-match predicate that
also requires UNAVAILABLE (so DATABASE_UNAVAILABLE and other transient
failures never select a mutation retry), and rebuild_with_message,
which rewrites a status message at the google.rpc.Status envelope
level so every error detail, known or unknown, survives message
appends.

The new create_retry module holds the pure retry policy shared by the
SDK and the CLI: a delay floor from the server's RetryInfo (1s when
absent), jitter capped at min(500ms, half the delay) derived from an
injected per-client seed, a wall-clock budget that stops instead of
overrunning, and the AlreadyExists disambiguation for named creates
(Deleting waits, NotFound resends the create up to three times, a live
holder is terminal).

Signed-off-by: Emilien Macchi <[email protected]>
…s in create compensation

A create that loses the SSH identity mutation lock budget returned a
bare UNAVAILABLE with no machine-readable reason, so clients could not
tell it apart from any other transient failure. Both identity lock
acquisitions now route PersistenceError::LockTimeout through
persistence_error_to_status, reporting the shared MUTATION_LOCK_TIMEOUT
reason and a 1s RetryInfo like the creation guard does; other
persistence failures keep the previous bare status.

compensate_failed_create rebuilt every status it returns with
Status::new, dropping the ErrorInfo and RetryInfo the create failure
carried and silently disabling client retry once cleanup appended to
the message. Every rebuild branch now goes through the core
rebuild_with_message helper, which rewrites the message at the
google.rpc.Status envelope level and keeps every error detail. The
begin-delete supersession arm now reports the original create failure
with the begin-delete outcome appended instead of replacing the status
outright, so the retry contract survives that branch too.

Signed-off-by: Emilien Macchi <[email protected]>
Add opt-in retrying entry points for all four create surfaces:
create_sandbox_with_retry and create_sandbox_from_template_with_retry
on both OpenShellClient and WorkspaceScopedClient, backed by one
shared loop over the core create-retry policy. SdkError gains
mutation_lock_timeout(), an exact-match predicate on the shared
reason; retryable() keeps its auth-only contract.

Each attempt waits at least the server's RetryInfo delay plus
per-call jitter and shares one wall-clock budget (CreateRetryOptions,
default 60s) with the AlreadyExists cleanup wait. request_id stays
empty on every attempt so mutation replay never surfaces a recorded
failure. A named create that hits AlreadyExists polls the holder's
phase in the create's workspace: a Deleting holder is waited out, a
missing record resends the create up to three times, and a live,
recovered, or different-id holder is a terminal conflict. Budget
exhaustion surfaces the last error annotated with the attempt count
and the original lock-timeout cause, keeping the raw status intact.

Signed-off-by: Emilien Macchi <[email protected]>
sandbox create now retries while the gateway mutation lock is busy,
using the same openshell-core decision layer as the SDK: each attempt
waits at least the server's RetryInfo delay plus per-invocation
jitter inside one wall-clock budget, and only MUTATION_LOCK_TIMEOUT
retries; every other failure may have persisted the create, so it
stays terminal.

The budget defaults to 60s, follows OPENSHELL_CREATE_RETRY_TIMEOUT,
and is set with --create-retry-timeout <SECONDS>; --no-create-retry
restores the fail-fast behavior. Progress lines go to stderr only, so
structured stdout output stays parseable. A named create that hits
AlreadyExists polls the holder's phase in the create's workspace: a
Deleting holder is waited out, a missing record resends the create up
to three times, and a live or recovered holder stops with the
delete-first hint. Budget exhaustion surfaces the last error
annotated with the attempt count and the original lock-timeout cause.

Signed-off-by: Emilien Macchi <[email protected]>
Document the create-retry contract end to end: the SDK's opt-in
retrying create methods and the mutation_lock_timeout accessor on the
api-errors page (with the retry conditions, delay floor, jitter bound,
budget, and the AlreadyExists cleanup-wait matrix), the automatic CLI
and Rust SDK retry on the high-availability page, the CLI flags and
cleanup-wait behavior in the openshell-cli skill, and the retrying
entry points in the SDK README. Correct the high-availability page
for the SSH identity step now carrying the MUTATION_LOCK_TIMEOUT
reason.

Closes NVIDIA#5697

Signed-off-by: Emilien Macchi <[email protected]>
@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@EmilienM

Copy link
Copy Markdown
Contributor Author

@FrostGod tagging you since you filed #5697. The PR takes it end to end: the server now keeps ErrorInfo/RetryInfo through failed-create cleanup (and the SSH identity step carries the reason), the SDK has opt-in retrying create methods, and the CLI retries by default within a bounded budget. The parts I'd most like your eyes on are the retry policy in openshell-core/src/create_retry.rs and the two client loops. One deliberate deviation from the issue: the budget is wall-clock with a 5-attempt cap rather than attempts-only, to bound lock-pool re-entry under saturation.

@FrostGod

Copy link
Copy Markdown
Collaborator

@EmilienM can we repurpose this PR to fix for other SDKs ( not rust or CLI ) if its okay

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: retry sandbox create on MUTATION_LOCK_TIMEOUT in the SDK and CLI

2 participants