Repository navigation
feat(services): publish authoritative application origins - #4956
TaylorMutch wants to merge 9 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4956.docs.buildwithfern.com/openshell |
a6d0833 to
e6e1927
Compare
932643d to
6c83b66
Compare
e6e1927 to
2574b9a
Compare
6c83b66 to
755e4d1
Compare
70e365e to
30cf9ce
Compare
0944ea2 to
8504067
Compare
30cf9ce to
6b2fe9c
Compare
8504067 to
feac6b4
Compare
dae4a9d to
66220a7
Compare
0939a70 to
cf5448a
Compare
66220a7 to
3dc01d7
Compare
433b9b9 to
35e2d05
Compare
fd8af36 to
29c0d05
Compare
35e2d05 to
c503a94
Compare
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 3. typescript-sdk — No explicit risk acceptance foundGenerated raw SandboxResponse, ServiceEndpointResponse, and SandboxServiceRoute types acquire required URL-map, URL, and applicationHostname properties absent from previously valid complete objects.
Review details
Advisory review; publication gates are unchanged. |
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. configuration — No explicit risk acceptance foundApp-listener TOML
2. api-behavior — No explicit risk acceptance found
Review details
Advisory review; publication gates are unchanged. |
|
The two changes in the compatibility advisory are intentional and explicitly accepted:
The PR compatibility notes now record this acceptance and the migration requirements. |
|
/ok to test 5500740 |
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. configuration — Risk explicitly acceptedApp-listener TOML/Helm
2. api-behavior — Risk explicitly accepted
Review details
Advisory review; publication gates are unchanged. |
There was a problem hiding this comment.
Thanks, the overall shape looks right. Sharing one hostname policy across URL responses, supervisor config, discovery, and SNI/Host removes a real source of drift. Having the gateway send application_hostname is the right call, because the hashed label depends on the gateway name, which the supervisor doesn't have.
I checked the routing toggle against a local Docker gateway at this head. With service_routing_enabled=false, the gateway returned 404 for every request I tried:
- plain HTTP
- WebSocket upgrade
- HTTP/2 prior knowledge
- uppercase and trailing-dot Host
- the app-domain Host
- an IP Host
- the loopback plaintext listener
base_url validation rejected http://, paths, IP literals, userinfo, trailing dots, and the legacy domain key. https://APPS.example.com:10443 was normalized to lowercase and kept its port. Hashed labels matched an independently computed SHA-256/base32 label.
Most of my comments are inline. The upgrade items are collected below because they cut across files.
Upgrades
1. A new gateway rejects installation reports from v0.1.9 supervisors, so running sandboxes drop out of discovery (should)
crates/openshell-server/src/grpc/service.rs:454 → crates/openshell-server/src/grpc/service_advertisement.rs:156
When the listener is enabled, configured_app_routes_from_endpoints now always sets application_hostname. A v0.1.9 supervisor decodes the route without field 6, so the routes it reports back don't include it (snapshot.routes.values().cloned() in the v0.1.9 supervisor). validate_routes compares routes with full struct equality (declared.contains(route)), so every report from an old supervisor fails with "installed service route is not a current declaration".
The result: after a Helm upgrade, a running sandbox whose pod still has the old supervisor disappears from discovery until the pod is recreated. The impact is narrow. It needs all of these:
- the app listener is enabled, which is opt-in and Kubernetes-only, and these operators already have to edit
domain→base_urlin this upgrade - the sandbox has at least one exposed service routed by hostname (sandboxes with no routes send empty reports, which still validate)
- the sandbox's pod still runs a v0.1.9 supervisor (dedicated-mode pods created before the upgrade; the shared DaemonSet and new sandboxes are fine)
Only discovery and app-listener ingress are affected; gateway_url routing keeps working. In hashed mode, the old supervisor also can't serve the hashed application_url, because it splits SNI on --. The legacy branch at service_discovery.rs:487 handles records stored by an older gateway, but not this direction.
Suggested fix: when the reported application_hostname is empty, compare the route with that field ignored. Then use the readable hostname for that installation, both in discovery and when choosing which application_url to publish. A test that sends a report with an empty application_hostname to a new gateway would cover it. If you'd rather not change the code, a line in the upgrade note ("recreate app-listener sandboxes after upgrading") would also cover it.
2. A new supervisor configured by an older driver fails at startup (should)
crates/openshell-supervisor/src/app_listener.rs:60
The supervisor now requires OPENSHELL_APP_BASE_URL. A driver from before this change sets only OPENSHELL_APP_SERVICE_DOMAIN, so from_values errors out and the sandbox or shared supervisor crash-loops. Helm usually rolls the driver and supervisor together, so this needs skew (a pinned supervisor image, or a DaemonSet that rolls first). A one-release fallback would turn it into a soft degrade: when OPENSHELL_APP_BASE_URL is absent, use https://<OPENSHELL_APP_SERVICE_DOMAIN>.
3. domain → base_url (acknowledged, no change requested)
Removing the v0.1.9 domain key is a deliberate, accepted break, and it fails loudly: the gateway refuses to start with "unknown field domain". I'm fine with it as is. Two small text fixes so operators and agents aren't misled:
docs/how-it-works/gateways/configuration.mdx:2111mentions apublic_urlsetting that never shipped. Saying "base_url = "https://<domain>"replacesdomain" would be enough. The test loop atcrates/openshell-core/src/app_listener.rs:288can droppublic_urltoo.skills/debug-openshell-cluster/SKILL.md:1737still says to checkportanddomain. An agent following the skill would write a config the gateway rejects.
4. Older CLIs rewrite the application port (FYI)
docs/how-it-works/gateways/configuration.mdx:2232 already says to upgrade the CLI alongside the gateway. That's worth carrying into the release notes together with item 3.
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
Signed-off-by: Taylor Mutch <[email protected]>
|
/ok to test f8d1af1 |
|
Following up on the upgrade notes in your review:
The six inline comments have code fixes and targeted regression coverage where applicable. Standalone and combined server tests, proxy tests, strict Clippy, Helm/runtime-parser checks, and the shared-listener k3d takeover/fallback E2E passed; PostgreSQL-dependent server tests were skipped. Mixed-version fallback support is outside the agreed scope. |
Summary
Publish authoritative application URLs from the configured supervisor application origin, independently of the gateway API origin. Return explicit application and gateway URLs so clients can choose an access path. Allow installations to disable gateway-mediated application routing independently.
Based on main at
df6a799eb.Related Issue
Closes #4953. Part of RFC 0017.
Changes
supervisor.app_listener.base_urlas the single authored application origin and derive its service domain. Enabled listeners require an HTTPS DNS base URL; replace the earlierdomainfield. The supervisor listener port stays independent of the public URL port.[openshell.gateway].enable_gateway_service_routing(defaulttrue). Setting it tofalseomits gateway URLs and returns404for gateway application HTTP/WebSocket traffic, while keeping control-plane APIs, app listeners, and discovery available.Configuration
Gateway routing is enabled by default. Disable it independently of the app listener using any of these equivalent settings:
[openshell.gateway]→enable_gateway_service_routing = false--enable-gateway-service-routing=falseOPENSHELL_ENABLE_GATEWAY_SERVICE_ROUTING=falsegatewayConfig."openshell.gateway".enable_gateway_service_routing: falsePrecedence is CLI > environment > TOML > default. Explicit
trueandfalseoverrides are supported.Example: publish only application-ingress URLs, with hashed hostnames:
base_urlsupplies the hostname suffix under which service DNS records are created. The listener derives its domain from that URL.namesupplies the stable installation identity for hashed names.service_hostname_formatdefaults to"readable"and applies to both ingress paths. The applicationbase_urlis an HTTPS origin whose hostname determines the service domain; its public port can differ from the listener port. DNS, certificates, and ingress are provisioned separately.For the routing opt-out alone in Helm:
Integrator perspective
The service API tells clients which public URL to use; discovery tells routers where to send requests.
base_urlfor your public ingress. Responses exposeapplication_url,gateway_urlwhen gateway routing is enabled, and a preferredurlthat selects the application URL when configured.Publishing a URL does not create DNS or ingress resources, or establish reachability. Both ingress paths use
[openshell.gateway].service_hostname_format; the app listener has no separate naming mode. Keeping the installation name and public HTTPS domain/port unchanged preserves the service URL when switching ingress.Public URL selection
This PR publishes the configured application origin alongside the gateway-mediated URL when enabled. The example uses gateway installation
gateway-a, workspacedefault, sandboxmy-sandbox, and serviceweb.flowchart TD I["Service identity<br/>gateway-a / default / my-sandbox / web"] M["One gateway setting: service_hostname_format<br/>readable or hashed"] I --> L["Shared service label generator<br/>Hashed example: enh47uqyj7bpyttz"] M --> L E["enable_gateway_service_routing<br/>Default: true"] --> G L --> G["gateway_url<br/>https://enh47uqyj7bpyttz.gateway.example.com/"] L --> A["application_url<br/>https://enh47uqyj7bpyttz.apps.example.com/"] O["Configured application public origin<br/>https://apps.example.com"] --> A A -->|"Preferred when configured"| U["url returned to clients"] G -->|"Fallback when application_url is absent"| USet
enable_gateway_service_routing = falseto use application ingress without advertising or serving gateway-mediated application URLs. The CLI/env equivalents are--enable-gateway-service-routing=falseandOPENSHELL_ENABLE_GATEWAY_SERVICE_ROUTING=false; Helm usesgatewayConfig."openshell.gateway".enable_gateway_service_routing: false.urlgateway_urlDisabling routing also makes
gateway_service_urlsempty. Idempotent retries replay the original URL bundle together, even if routing has since been disabled. Get/List return current addresses.Hashed mode is shown above. Readable mode uses
<workspace>--<sandbox>[--<service>]for both paths. The two access paths are alternatives; when their public origins match, their complete URLs match. Integrators consume discovery hostnames as returned.Connecting the published URL to a sandbox
The proxy and example controller below are alternative discovery consumers, shown for context. This PR publishes URLs; it does not deploy either integration. Dashed arrows configure routing; solid arrows carry application traffic.
flowchart TD API["Service API<br/>Returns public application URL"] -.->|"URL used by client"| Client["Application client"] D["Discovery API<br/>Exact hostname → current supervisor Pod IP:port<br/>Installation identity and expiry"] D -.->|"Snapshots / watches"| P["Sandbox proxy<br/>TLS passthrough"] D -.->|"Snapshots / watches"| C["Example Kubernetes HTTPRoute controller<br/>Separate POC"] C -.->|"Reconciles HTTPRoutes, backend resources<br/>and backend TLS configuration"| K["Kubernetes Gateway<br/>Edge TLS termination"] Client -->|"Option A: HTTPS to published hostname"| P Client -->|"Option B: HTTPS to published hostname"| K P -->|"Application TLS forwarded unchanged"| S["Owning supervisor<br/>Application TLS listener"] K -->|"HTTPS to discovered endpoint<br/>Preserve application Host and backend SNI"| S S --> W["Declared service inside sandbox"]Both paths keep application traffic out of the OpenShell gateway. The integrator provides DNS, certificates, and ingress configuration; discovery supplies backend changes and withdrawal/expiry information.
Release Notes
Publish application URLs independently of the gateway connection. Gateway-mediated application routing can be disabled with
enable_gateway_service_routing(runtime defaulttrue). When upgrading application ingress, replacedomainwithbase_url, deploy matching gateway/driver/supervisor images, recreate old dedicated sandbox Pods with the updated supervisor image, and upgrade the CLI.Compatibility context
The two changes in the compatibility advisory are intentional and explicitly accepted:
domainwithbase_url, and the internal supervisor environment variable withOPENSHELL_APP_BASE_URL, is accepted without legacy aliases. Remove the old TOML keys and deploy matching gateway/supervisor images.Enabled application listeners now require
base_urlin place ofdomain. Remove the old key when upgrading and deploy matching gateway/driver/supervisor images. The domain is derived from the HTTPS base URL; its public port remains separate from the supervisor listener port.Supervisor route declarations are an internal, version-locked deployment protocol. Recreate existing dedicated sandbox Pods with the matching supervisor image when upgrading application ingress: older supervisors cannot acknowledge the new hostname declarations, so their services drop out of discovery until replaced; they also cannot serve hashed application hostnames. Plan for service interruption during replacement. New supervisors require
OPENSHELL_APP_BASE_URLfrom the driver. Mixed-version compatibility fallbacks are intentionally not provided. Updated supervisors retain readable listener aliases.Protobuf additions remain additive on the wire. Curated SDK callers retain their existing construction paths; new TypeScript URL maps are optional. Hand-written exhaustive generated Rust/TypeScript message literals must include new fields or use generated/default initialization; preserving those literal shapes is outside this change's migration scope.
Python SDK changes are deferred to the external SDK repository; this PR leaves the frozen core SDK files unchanged.
Testing
df6a799eb, preserving the SPIFFE provider-token limiter and application-auth peer metadata. Branch ancestry, patch scope, and frozen Python/Go SDK checks passed.enable_gateway_service_routing: true. The isolated release and namespace were removed.Dedicated, poll, abrupt-loss, and external-ingress scenarios were not rerun for these review fixes. Current-head GitHub CI is pending.
Qualification limits
Live Kubernetes qualification covers dedicated/shared supervisors, graceful shared-owner replacement, gateway outage, and policy isolation. Hashed URL/public-port behavior is covered by unit tests. Whole-node failure, real network partitions, metric-driven HPA, sustained production load, external ingress/DNS automation, and published release artifacts remain unqualified. Installation-status presentation (#4670) remains separate work. The example Kubernetes HTTPRoute controller remains a separate compatibility POC.
Checklist
PR dependency order: #4956 → #4963 → #4964 → #4957.