Skip to content

fix(extensions): resend calls an extension service did not process - #5735

Open
dvavili wants to merge 1 commit into
NVIDIA:mainfrom
dvavili:fix/5713-extension-call-retry/dvavili
Open

dvavili wants to merge 1 commit into
NVIDIA:mainfrom
dvavili:fix/5713-extension-call-retry/dvavili

Conversation

@dvavili

@dvavili dvavili commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

A gateway interceptor or supervisor middleware call fails closed when its connection closes as an extension service (like Agent Policy Fabric) restarts, even when a replacement instance is already serving. This PR resends a unary extension call when the transport reports the service did not process it.

Related Issue

Closes #5713

Changes

  • openshell-extension-core: add call_was_not_processed, which recognises a connect failure, a request cancelled before it was sent, a graceful GOAWAY, and a refused stream, and call_with_retries, which resends such a call with delays from 5 ms to 100 ms for up to 5 seconds.
  • openshell-gateway-interceptors: Evaluate, SnapshotProviderProfiles and Describe go through call_with_retries, inside the binding's existing timeout.
  • openshell-supervisor-middleware: EvaluateHttpRequest, Describe and ValidateConfig do the same.
  • A call the service may have processed is not resent, and streaming calls are unchanged.
  • Docs: gateway interceptors and supervisor middleware operations pages describe the behavior.

Release Notes

Gateway interceptor and supervisor middleware calls no longer fail when an extension service is replaced by a rolling restart. A call that the service did not process is sent again within the call's timeout.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

cargo fmt --all -- --check, cargo clippy --all-targets -- -D warnings and cargo test pass for the three changed crates.

Also run on a kind cluster against a build without this change, with egress requests and sandbox creations in loops while an extension service that serves an interceptor and a middleware is restarted:

Scenario Traffic Without this change With this change
Rolling update, new instance ready before the old one stops (40 restarts) Egress requests 5 of 98,974 failed 0 of 99,179 failed
Rolling update, new instance ready before the old one stops (40 restarts) Sandbox creations 2 of 15,563 failed 0 of 15,654 failed
One of two instances killed without a grace period (20 kills) Egress requests 1 of 31,412 failed 0 of 31,599 failed
One of two instances killed without a grace period (20 kills) Sandbox creations 3 of 4,951 failed 0 of 4,929 failed
Single instance down about 9 s, less than the 10 s middleware timeout (10 restarts) Egress requests 164 of 15,113 failed 10 of 14,937 failed
Single instance down about 9 s, longer than the 500 ms interceptor timeout (10 restarts) Sandbox creations 149 of 2,487 failed 128 of 2,452 failed

The table counts calls that failed in the transport. When the service is down for longer than the call's timeout, calls still fail closed, as the last row shows.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

A gateway interceptor or supervisor middleware call failed closed when
its connection to the extension service closed as the service
restarted, even with a replacement instance already serving.

Unary extension calls are now sent again when the transport reports
that the service did not process them: a connect failure, a request
cancelled before it was sent, a graceful GOAWAY, or a refused stream.
Attempts back off from 5 ms to 100 ms for up to 5 seconds and stay
inside the call's own timeout. A call the service may have processed
is not sent again.

Closes NVIDIA#5713

Signed-off-by: Divya Vavili <[email protected]>
@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: extension calls fail closed while a gateway interceptor or supervisor middleware service restarts

1 participant