Skip to content

feat(observability): supervisor decision and latency metrics - #5720

Draft
aegeiger wants to merge 4 commits into
NVIDIA:mainfrom
aegeiger:feat/5364-supervisor-metrics/aegeiger
Draft

aegeiger wants to merge 4 commits into
NVIDIA:mainfrom
aegeiger:feat/5364-supervisor-metrics/aegeiger

Conversation

@aegeiger

Copy link
Copy Markdown
Contributor

Summary

Add opt-in OTLP export for supervisor egress metrics. Operators can alert on denial rates, see which stage is adding latency, and track load across dedicated and shared supervisors

Related Issue

closes #5364

Changes

  • Count final egress admission decisions at connection, request, and message scope with bounded policy labels.
  • Measure policy evaluation, request and message inspection, and upstream connect latency. All three histograms use the same result vocabulary and include sub-millisecond buckets.
  • Report active proxied connections and assigned sandboxes. Use OTLP resource attributes to identify the gateway and supervisor process.
  • Reuse the shared OTLP exporter and document collector configuration, metric labels, and example PromQL queries.

Release Notes

Set enabled = true under [openshell.gateway.otlp.supervisor_metrics] to export supervisor metrics. The table can use its own collector or inherit the gateway’s shared OTLP collector settings.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Count connection admissions at the final decision, including the
middleware gate for SQL passthrough. Bound policy-name labels and
distinguish quarantine denials from requests that matched no policy.

Measure policy evaluation and direct or proxied dials separately, with
sub-millisecond buckets for local work. Expose active proxied connections
by inspection type and classify upstream proxy authentication failures
as connect errors.

Signed-off-by: Eitan Geiger <egeiger@redhat.com>
Record one final admission per inspected HTTP request or WebSocket text
message. An enforced denial takes precedence over fail-open, which takes
precedence over audit.

Time inspection up to forwarding or a terminal result, excluding
destination lookup and dial time. Classify a failed forward dial as an
error or timeout and malformed WebSocket messages as inspection errors.

Signed-off-by: Eitan Geiger <egeiger@redhat.com>
Add an opt-in supervisor_metrics collector to gateway TOML. Inherit the
base OTLP collector settings when the table omits an endpoint, or use
the table's collector settings when it specifies one.

Pass endpoints through supervisor launch settings and credentials
through the trusted auth bundle. Send equivalent settings to shared
supervisors on the authenticated assignment stream and omit the
credential-bearing payload from debug formatting.

Signed-off-by: Eitan Geiger <egeiger@redhat.com>
Start an OTLP metrics provider in each configured supervisor runtime.
Attach the gateway name and a per-process instance ID as resource
attributes for grouping metrics by gateway and supervisor.

Register the network instruments and report sandbox assignments with a
gauge. Clear the assignment count before flushing on shutdown. Document
the gateway opt-in, metric catalog, and admission semantics.

Signed-off-by: Eitan Geiger <egeiger@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@aegeiger
aegeiger marked this pull request as draft November 18, 2026 20:24
@krishicks

Copy link
Copy Markdown
Collaborator

Drafted by 🤖, signed off by @krishicks

Config: please drop [openshell.gateway.otlp.supervisor_metrics] and have [openshell.gateway.otlp.metrics] cover supervisor metrics too. Each existing signal table already serves both the gateway and the supervisors. [otlp.traces] and [otlp.ocsf] set export for both, and supervisors get the same endpoint, protocol, headers and TLS through the auth bundle or the assignment stream. Metrics should work the same way: enabled = true under [otlp.metrics] exports gateway metrics and supervisor metrics to the same collector. With a separate table, an operator who enables [otlp.metrics] would reasonably expect all OpenShell metrics and get only the gateway's. As with traces and OCSF, the collector has to be reachable from supervisors.

Supervisor identity: service.instance.id is a new random UUID per process, so every restart starts a new series set. The Prometheus OTLP receiver turns service.instance.id into the instance label, so that churn shows up on every metric. Use an ID that is stable across restarts, or leave the ID out. A deterministic ID derived from inputs that survive a restart would be better than leaving it out, for example the gateway's ID combined with:

  • the pool member ID (PoolIdentity.member_id) for shared supervisors, which the gateway already knows;
  • the sandbox ID for dedicated supervisors.

The same supervisor then keeps the same ID after a restart, and the ID can be traced back to what the gateway reports. Dedicated supervisors' metrics should also carry openshell.sandbox.id, the same as their traces and OCSF records do through supervisor_resource_attributes.

Exemplars: none are produced. Timer captures and re-attaches the OTel Context (metrics.rs:361, metrics.rs:395), but opentelemetry_sdk 0.33.1 never fills in exemplars: its histogram aggregation always emits exemplars: vec![]. #5364 lists exemplars as an acceptance criterion, and there is no test for them. Either attach exemplars another way and test that a sampled span produces one, or remove the context plumbing and track exemplars separately.

Collector credentials on the assignment stream: HostAccepted.supervisor_metrics (supervisor_host.proto:100) carries collector headers and TLS client keys. Unlike the other credential fields in that file, including launch_material, it isn't marked [(openshell.options.v1.secret) = true]. It relies on a hand-written Debug impl and skip_debug instead (proto/mod.rs:90). Mark it secret. With the config change above, this field could also become a typed message that holds the shared export settings, instead of JSON bytes.

policy label: the label keeps the first 16 policy names each process sees (metrics.rs:261). A shared supervisor serves many sandboxes, so:

  • policies with the same name in different sandboxes merge into one series;
  • whichever sandboxes connect first fill the 16 slots;
  • the same policy can be named on one supervisor and overflow on another.

That makes sum by (outcome, policy) misleading, and it's the example query in the docs. Either drop the label, as the original proposal on the issue did, or find a bound that gives the same answer on every supervisor.

Shared vocabulary: the PR adds new fields instead of reusing existing ones. Inspected request spans now carry openshell.egress.outcome=deny next to the existing openshell.request.outcome=denied (request_trace.rs:76). Every inspected OCSF HTTP record also gains unmapped.egress_inspection_type, egress_result and egress_outcome (correlation.rs:112). That changes what SIEM consumers receive, and the PR description doesn't mention it. Metric labels should use the values spans and OCSF already emit, not a second set of fields carrying the same information.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): supervisor decision and latency metrics

2 participants