Skip to content

test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance - #5665

Open
matthewgrossman wants to merge 25 commits into
mainfrom
test/5655-port-network-l7-policy/matthewgrossman
Open

matthewgrossman wants to merge 25 commits into
mainfrom
test/5655-port-network-l7-policy/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Port the network, L7, policy-update, and port-forward tests from e2e/rust to testkit, and delete the legacy files. Most tests move into the conformance suite. The IP-literal egress test moves to a new routable-host-ip feature package.

Stack: main ← this PR ← #5695. #5691 (shared sandbox client helpers) has merged into main, and this branch is rebased onto it.

Related Issue

Closes #5655
Part of #5281

Follow-up: #5752 (the k3s-ha skip below).

Changes

  • testkit: Context::compute_driver, SandboxBuilder::driver_config, Sandbox::logs / logs_until, policy::set, policy::from_yaml, and policy::base_with.
  • host-server support: handler responses, chunked bodies, a raw TcpServer, free_port, and a loopback exchange client.
  • Shared helpers from test(testkit): share sandbox client helpers across testkit suites #5691: the suites use testkit's Sandbox::run_python, ExecOutput::last_json, Sandbox::policy_revisions / active_policy, SandboxBuilder::providers and policy::binaries_yaml, plus host_server_support::TEST_HOST. Conformance no longer has a support module.
  • Probes instead of one-off scripts: the bypass, reserved-destination, status, REST, REST upgrade, echo, uninspectable, redaction, pipeline and credential rewrite clients are now sandbox_clients_support::{connect, http, http_raw} calls. Credential tests read the provider placeholder with printenv and send it with http.
  • Scripts moved to tests/support/sandbox-clients/scripts: the GraphQL, JSON-RPC, JSON-RPC audit, MCP, localhost, listener lifetime, preconnect owner, socket inheritance, reload keep-alive, echo server, WebSocket text and binary, and bound credential (bash) clients, exposed through sandbox_clients_support::scripts. routable-host-ip no longer includes a conformance script from another workspace.
  • New conformance suites:
    • network: bypass denial, reserved destinations, localhost without proxy env, listener lifetime.
    • network_l7 + network_l7/mcp: REST, GraphQL, JSON-RPC, and MCP.
    • network_egress: reload, ambiguous update, tls: skip, middleware, pipelining.
    • policy_update and port_forward.
  • credential_gating additions: header and body rewrite, WebSocket text rewrite, and static credentials bound to profile endpoints.
  • New feature package tests/suites/features/routable-host-ip: the IP-literal endpoint test (explicit allowed_ips and implicit). Policy DNS gives the sandbox a synthetic address for host.openshell.internal, so the test reaches a host server by name first and uses the address the request arrived at (host-server Request::local_addr). CI rows: Docker rootful (deb), Podman rootful (rpm), k3s, and k3s-ha.
  • Denial codes: denied L7, GraphQL, JSON-RPC and pipelined requests must return policy_denied, and REST upgrades and the fail-closed middleware unsupported_l7_protocol, as documented under "Proxy Error Responses".
  • k3s-ha skip: a log read on a multi-replica gateway can reach a replica without the sandbox's logs (bug: sandbox logs and watch streams ignore gateway ownership in multi-replica mode #5462). The k3s-ha setup publishes OPENSHELL_TEST_GATEWAY_REPLICAS=3, and the two tests that read logs (profile_endpoints_bind_static_credentials, fail_closed_middleware_blocks_uninspectable_traffic) skip themselves there. test(conformance): run the two sandbox-log tests on k3s-ha again #5752 tracks removing this.
  • Push mode: live_policy_update is removed from e2e:rust:push. That lane already runs openshell-conformance against a push-mode gateway, so it runs the policy_update tests instead. The tmachine config-push environment is left to test(conformance): run the conformance suite against a gateway that pushes configuration #5622.
  • Removed:
    • The 11 legacy files, plus static_provider_credentials_are_bound_to_profile_endpoints and its helpers from host_gateway_alias.rs. The other two tests in that file stay until PR5.
    • The matching [[test]] entries, PODMAN_CI_TESTS entries, and artifacts.nix follow-up binary.
    • The e2e:websocket-conformance task and the now-unused SandboxGuard::spawn_forward.
  • Docs: conformance README suites table, TESTING.md (routable-host-ip), and a note on Context::name that sandbox suffixes can have at most 5 characters (test(testkit): share sandbox client helpers across testkit suites #5691's MAX_SANDBOX_NAME_LEN check enforces it).

Defaults applied

  • Upstream fixtures are in-process Rust servers on the test host, reached at host.openshell.internal. The legacy private-range allowed_ips entries are gone.
  • Client scripts run through Sandbox::exec in sleep infinity sandboxes, using pipes. The legacy tests were also non-TTY, so PTY/pipe parity is unchanged.
  • There is no exec_until helper; Sandbox::poll_exec already covers it.
  • Network tests use policy::base(), python_image(), and the shared Python binaries list.
  • Merges keep every assertion:
    • REST allow and deny are merged.
    • The JSON-RPC audit-mode case is a second endpoint in the JSON-RPC test.
    • The JSON-RPC test includes the REST-route and unlisted-method cases fix(policy): align request authorization with route protocol #5274 added to the legacy test after this port began.
    • MCP is split into three tests, one per protocol version set.
    • The live_policy_update round-trip and from-empty tests are merged into policy_set_versions_and_ignores_repeats.
  • Policy-update tests wait until the active revision is loaded and none is pending or failed. This covers the supervisor's enrichment of the base policy.

Deviations from the plan

  • port_forward is its own module, not part of cli.rs, to keep it separate from the CLI tests. It starts forward start -d with cli_spawn and waits for the exit code, because the backgrounded ssh keeps the CLI's pipes open. A cleanup hook runs forward stop.
  • The static credential binding test lives in credential_gating.rs. The unbound endpoint is a second port on the test host instead of host.docker.internal, so the 403 holds on every driver. The legacy Podman connect-denied special case is gone.

Rows that kept a container fixture

None. The IP-literal test needs a host address that the supervisor allows as an IP literal. Rootless Podman's address is link-local, which is always blocked. So instead of keeping a container fixture, that test is a feature package that does not run on rootless Podman.

Driver coverage changes

  • Docker and Kubernetes ran all of these tests before and still do, through the conformance rows.
  • Podman previously ran only bypass_detection, forward_proxy_*, host_gateway_alias, live_policy_update, and no_proxy. It now runs all of the ported conformance tests, on rootful and rootless.
  • The VM host lane previously ran only host_gateway_alias from this set. It now runs all of them through openshell-conformance in e2e-vm.sh. That is new coverage. If a newly covered row fails, I'll raise it here before skipping anything.

Testing

  • cargo fmt --check and cargo clippy --locked --all-targets -D warnings for tests/testkit, tests/support, tests/suites/conformance, tests/suites/features
  • cargo test for tests/testkit and tests/support (new host-server unit tests included)
  • tasks/scripts/check-conformance-program.sh (77 tests)
  • cargo check --tests for e2e/rust with docker, podman, kubernetes, vm, and config-push features. cargo clippy on host_gateway_alias is clean.
  • E2E suites (CI)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 858dbc6

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 858dbc6. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 45fd589

@matthewgrossman
matthewgrossman force-pushed the test/5655-port-network-l7-policy/matthewgrossman branch from 858dbc6 to 45fd589 Compare November 18, 2026 05:10
@matthewgrossman
matthewgrossman changed the base branch from test/5426-port-cli-attach-exec/matthewgrossman to test/5689-share-sandbox-helpers/matthewgrossman November 18, 2026 06:23
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 251835b

@matthewgrossman
matthewgrossman force-pushed the test/5689-share-sandbox-helpers/matthewgrossman branch from b76ff7b to 6eae90b Compare November 18, 2026 18:12
@matthewgrossman
matthewgrossman force-pushed the test/5655-port-network-l7-policy/matthewgrossman branch from 251835b to 3214f39 Compare November 18, 2026 18:34
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 3214f39

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test f99bacb

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test afcf2f4

@matthewgrossman
matthewgrossman force-pushed the test/5655-port-network-l7-policy/matthewgrossman branch from f99bacb to afcf2f4 Compare November 18, 2026 23:27
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 054ed4f

… tests

These tests now run in the conformance suite and the routable-host-ip
feature package.

Signed-off-by: Matthew Grossman <[email protected]>
…er limit

The gateway rejects sandbox names longer than 19 characters, which leaves
5 for a suffix after test-<run-id>-. The static credential binding test
also needs the Python workload image, which has the sandbox user that the
base policy runs as.

Signed-off-by: Matthew Grossman <[email protected]>
Drops the include of a conformance client script from another workspace.

Signed-off-by: Matthew Grossman <[email protected]>
Replace conformance's support module and generic client scripts with
testkit's run_python, last_json, policy readers, providers and
binaries_yaml, and the connect, http and http_raw probes. Protocol and
specialized clients move to the sandbox-clients scripts module.

The pipelined request tests now count every response in the reply; the
403 follows the first response's body with no line break, so the old
line-based count missed it. The policy update tests use the Python
image, which has the sandbox user the base policy runs as.

Signed-off-by: Matthew Grossman <[email protected]>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 31e7182

@matthewgrossman
matthewgrossman force-pushed the test/5655-port-network-l7-policy/matthewgrossman branch from 054ed4f to 31e7182 Compare November 19, 2026 00:46
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b82757c

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test bfcbcc7

@matthewgrossman
matthewgrossman marked this pull request as ready for review November 19, 2026 06:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance

1 participant