Repository navigation
Conversation
A gateway interceptor or supervisor middleware call failed closed when its connection to the extension service closed as the service restarted, even with a replacement instance already serving. Unary extension calls are now sent again when the transport reports that the service did not process them: a connect failure, a request cancelled before it was sent, a graceful GOAWAY, or a refused stream. Attempts back off from 5 ms to 100 ms for up to 5 seconds and stay inside the call's own timeout. A call the service may have processed is not sent again. Closes NVIDIA#5713 Signed-off-by: Divya Vavili <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A gateway interceptor or supervisor middleware call fails closed when its connection closes as an extension service (like Agent Policy Fabric) restarts, even when a replacement instance is already serving. This PR resends a unary extension call when the transport reports the service did not process it.
Related Issue
Closes #5713
Changes
openshell-extension-core: addcall_was_not_processed, which recognises a connect failure, a request cancelled before it was sent, a gracefulGOAWAY, and a refused stream, andcall_with_retries, which resends such a call with delays from 5 ms to 100 ms for up to 5 seconds.openshell-gateway-interceptors:Evaluate,SnapshotProviderProfilesandDescribego throughcall_with_retries, inside the binding's existing timeout.openshell-supervisor-middleware:EvaluateHttpRequest,DescribeandValidateConfigdo the same.Release Notes
Gateway interceptor and supervisor middleware calls no longer fail when an extension service is replaced by a rolling restart. A call that the service did not process is sent again within the call's timeout.
Testing
cargo fmt --all -- --check,cargo clippy --all-targets -- -D warningsandcargo testpass for the three changed crates.Also run on a kind cluster against a build without this change, with egress requests and sandbox creations in loops while an extension service that serves an interceptor and a middleware is restarted:
The table counts calls that failed in the transport. When the service is down for longer than the call's timeout, calls still fail closed, as the last row shows.
Checklist