Repository navigation
Conversation
…sion layer Add the shared primitives that let clients retry sandbox creates that lose the gateway mutation lock budget. rpc_error gains the MUTATION_LOCK_TIMEOUT reason constant, an exact-match predicate that also requires UNAVAILABLE (so DATABASE_UNAVAILABLE and other transient failures never select a mutation retry), and rebuild_with_message, which rewrites a status message at the google.rpc.Status envelope level so every error detail, known or unknown, survives message appends. The new create_retry module holds the pure retry policy shared by the SDK and the CLI: a delay floor from the server's RetryInfo (1s when absent), jitter capped at min(500ms, half the delay) derived from an injected per-client seed, a wall-clock budget that stops instead of overrunning, and the AlreadyExists disambiguation for named creates (Deleting waits, NotFound resends the create up to three times, a live holder is terminal). Signed-off-by: Emilien Macchi <[email protected]>
…s in create compensation A create that loses the SSH identity mutation lock budget returned a bare UNAVAILABLE with no machine-readable reason, so clients could not tell it apart from any other transient failure. Both identity lock acquisitions now route PersistenceError::LockTimeout through persistence_error_to_status, reporting the shared MUTATION_LOCK_TIMEOUT reason and a 1s RetryInfo like the creation guard does; other persistence failures keep the previous bare status. compensate_failed_create rebuilt every status it returns with Status::new, dropping the ErrorInfo and RetryInfo the create failure carried and silently disabling client retry once cleanup appended to the message. Every rebuild branch now goes through the core rebuild_with_message helper, which rewrites the message at the google.rpc.Status envelope level and keeps every error detail. The begin-delete supersession arm now reports the original create failure with the begin-delete outcome appended instead of replacing the status outright, so the retry contract survives that branch too. Signed-off-by: Emilien Macchi <[email protected]>
Add opt-in retrying entry points for all four create surfaces: create_sandbox_with_retry and create_sandbox_from_template_with_retry on both OpenShellClient and WorkspaceScopedClient, backed by one shared loop over the core create-retry policy. SdkError gains mutation_lock_timeout(), an exact-match predicate on the shared reason; retryable() keeps its auth-only contract. Each attempt waits at least the server's RetryInfo delay plus per-call jitter and shares one wall-clock budget (CreateRetryOptions, default 60s) with the AlreadyExists cleanup wait. request_id stays empty on every attempt so mutation replay never surfaces a recorded failure. A named create that hits AlreadyExists polls the holder's phase in the create's workspace: a Deleting holder is waited out, a missing record resends the create up to three times, and a live, recovered, or different-id holder is a terminal conflict. Budget exhaustion surfaces the last error annotated with the attempt count and the original lock-timeout cause, keeping the raw status intact. Signed-off-by: Emilien Macchi <[email protected]>
sandbox create now retries while the gateway mutation lock is busy, using the same openshell-core decision layer as the SDK: each attempt waits at least the server's RetryInfo delay plus per-invocation jitter inside one wall-clock budget, and only MUTATION_LOCK_TIMEOUT retries; every other failure may have persisted the create, so it stays terminal. The budget defaults to 60s, follows OPENSHELL_CREATE_RETRY_TIMEOUT, and is set with --create-retry-timeout <SECONDS>; --no-create-retry restores the fail-fast behavior. Progress lines go to stderr only, so structured stdout output stays parseable. A named create that hits AlreadyExists polls the holder's phase in the create's workspace: a Deleting holder is waited out, a missing record resends the create up to three times, and a live or recovered holder stops with the delete-first hint. Budget exhaustion surfaces the last error annotated with the attempt count and the original lock-timeout cause. Signed-off-by: Emilien Macchi <[email protected]>
Document the create-retry contract end to end: the SDK's opt-in retrying create methods and the mutation_lock_timeout accessor on the api-errors page (with the retry conditions, delay floor, jitter bound, budget, and the AlreadyExists cleanup-wait matrix), the automatic CLI and Rust SDK retry on the high-availability page, the CLI flags and cleanup-wait behavior in the openshell-cli skill, and the retrying entry points in the SDK README. Correct the high-availability page for the SSH identity step now carrying the MUTATION_LOCK_TIMEOUT reason. Closes NVIDIA#5697 Signed-off-by: Emilien Macchi <[email protected]>
Contributor
Author
|
@FrostGod tagging you since you filed #5697. The PR takes it end to end: the server now keeps ErrorInfo/RetryInfo through failed-create cleanup (and the SSH identity step carries the reason), the SDK has opt-in retrying create methods, and the CLI retries by default within a bounded budget. The parts I'd most like your eyes on are the retry policy in openshell-core/src/create_retry.rs and the two client loops. One deliberate deviation from the issue: the budget is wall-clock with a 5-attempt cap rather than attempts-only, to bound lock-pool re-entry under saturation. |
Collaborator
|
@EmilienM can we repurpose this PR to fix for other SDKs ( not rust or CLI ) if its okay |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sandbox createand the Rust SDK's create methods now retry a create that loses the gateway mutation lock, so burst creation under contention no longer needs a retry loop in every caller. The gateway keeps the machine-readable retry signal when failed-create cleanup rewrites the error, and both lock-timeout producers carry it.Related Issue
Closes #5697
Changes
LockTimeouttoUNAVAILABLE+MUTATION_LOCK_TIMEOUT+RetryInfo, andcompensate_failed_createpreservesErrorInfo/RetryInfoon every rebuilt statustonic::Status, a detail-preserving rebuild helper, and the retry policy itself (delay floor with jitter, backoff, 5-attempt cap, wall-clock budget, AlreadyExists phase decisions), shared by the SDK and the CLIcreate_sandbox_with_retry/create_sandbox_from_template_with_retryon both clients,CreateRetryOptionswith a 60s default budget, andSdkError::mutation_lock_timeout();retryable()keeps its auth-only contract--create-retry-timeout,--no-create-retry, andOPENSHELL_CREATE_RETRY_TIMEOUT(60s default, 24h clamp); progress on stderr only; a named create waits out a holder that is deleting or still provisioning; a--fromcreate re-stages its archive when the gateway reports the staging token spentRelease Notes
sandbox createnow retries for up to 60 seconds when the gateway mutation lock is busy and waits out a same-name sandbox that is still being deleted. The Rust SDK gains opt-in retrying create methods.MUTATION_LOCK_TIMEOUTfailures keep their reason and retry delay even after failed-create cleanup rewrites the message.Testing
Tests cover the reason predicate, detail survival through every compensation branch, retry timing (floor, jitter, backoff, attempt cap), budget exhaustion, the AlreadyExists phase matrix (deleting, provisioning, missing, live, different-id holder), staging-token re-staging, and deadline-bounded holder polls. The PostgreSQL lock suite passed 48/48. The one local failure is a pre-existing z3 solver timeout in
openshell-prover, reproduced on cleanorigin/main. No e2e lane: this is client retry policy plus server error mapping, covered by the SDK/CLI mock suites and the server compensation tests.Checklist