Skip to content

feat(sdk): retry sandbox create on MUTATION_LOCK_TIMEOUT - #5725

Open
ericcurtin wants to merge 1 commit into
NVIDIA:mainfrom
ericcurtin:feat/5697-create-retry/ericcurtin
Open

ericcurtin wants to merge 1 commit into
NVIDIA:mainfrom
ericcurtin:feat/5697-create-retry/ericcurtin

Conversation

@ericcurtin

Copy link
Copy Markdown
Contributor

Summary

Retry sandbox create in the Rust SDK and CLI when the gateway reports MUTATION_LOCK_TIMEOUT.

Related Issue

Closes #5697

Changes

  • Core: shared RetryPolicy and retry_create (server delay floor, jitter, time budget).
  • SDK: SdkError::is_lock_contention, retryable(), with_create_retry.
  • CLI: --create-retry-timeout, --no-create-retry.
  • Server: failed-create cleanup keeps ErrorInfo/RetryInfo; a name held by a Deleting sandbox returns SANDBOX_DELETING, which clients wait out only after a failed attempt.
  • Docs and CLI skill updated.

Release Notes

openshell sandbox create and the Rust SDK now retry for up to 60s when the gateway is busy.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Ran clippy and tests for openshell-core, openshell-sdk, openshell-cli and the server compute tests.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ericcurtin

Copy link
Copy Markdown
Contributor Author

@drew @shiju-nv PTAL when you get a chance. /ok to test aabb51a03c3e1f8490b8fd061c40303161dc92d2 if it looks good. Thank you!

@FrostGod

Copy link
Copy Markdown
Collaborator

@ericcurtin @EmilienM , won't it be better to have this retry in other SDKs too ?, Python, Typescript etc, to have a consistent behaviour

@EmilienM

Copy link
Copy Markdown
Contributor

I was also working on this issue, I'll post the PR. Happy to close mine and keep yours.

@FrostGod

Copy link
Copy Markdown
Collaborator

@EmilienM , we can take this PR for Rust & CLI, and can we repurpose this PR to fix for other SDKs if its okay

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: retry sandbox create on MUTATION_LOCK_TIMEOUT in the SDK and CLI

3 participants