Repository navigation
test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance - #5665
matthewgrossman wants to merge 25 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-5665.docs.buildwithfern.com/openshell |
|
/ok to test 858dbc6 |
|
Label |
|
/ok to test 45fd589 |
858dbc6 to
45fd589
Compare
|
/ok to test 251835b |
b76ff7b to
6eae90b
Compare
251835b to
3214f39
Compare
|
/ok to test 3214f39 |
|
/ok to test f99bacb |
3214f39 to
f99bacb
Compare
|
/ok to test afcf2f4 |
f99bacb to
afcf2f4
Compare
|
/ok to test 054ed4f |
…olicy set helpers Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
…ewrite tests Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
… tests These tests now run in the conformance suite and the routable-host-ip feature package. Signed-off-by: Matthew Grossman <[email protected]>
…suites Signed-off-by: Matthew Grossman <[email protected]>
…er limit The gateway rejects sandbox names longer than 19 characters, which leaves 5 for a suffix after test-<run-id>-. The static credential binding test also needs the Python workload image, which has the sandbox user that the base policy runs as. Signed-off-by: Matthew Grossman <[email protected]>
Drops the include of a conformance client script from another workspace. Signed-off-by: Matthew Grossman <[email protected]>
Replace conformance's support module and generic client scripts with testkit's run_python, last_json, policy readers, providers and binaries_yaml, and the connect, http and http_raw probes. Protocol and specialized clients move to the sandbox-clients scripts module. The pipelined request tests now count every response in the reply; the 403 follows the first response's body with no line break, so the old line-based count missed it. The policy update tests use the Python image, which has the sandbox user the base policy runs as. Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
…denials Signed-off-by: Matthew Grossman <[email protected]>
… the finding name Signed-off-by: Matthew Grossman <[email protected]>
|
/ok to test 31e7182 |
054ed4f to
31e7182
Compare
Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
…e-host-ip Signed-off-by: Matthew Grossman <[email protected]>
|
/ok to test b82757c |
…ca gateways Signed-off-by: Matthew Grossman <[email protected]>
|
/ok to test bfcbcc7 |
Summary
Port the network, L7, policy-update, and port-forward tests from
e2e/rustto testkit, and delete the legacy files. Most tests move into the conformance suite. The IP-literal egress test moves to a newroutable-host-ipfeature package.Stack:
main← this PR ← #5695. #5691 (shared sandbox client helpers) has merged intomain, and this branch is rebased onto it.Related Issue
Closes #5655
Part of #5281
Follow-up: #5752 (the k3s-ha skip below).
Changes
Context::compute_driver,SandboxBuilder::driver_config,Sandbox::logs/logs_until,policy::set,policy::from_yaml, andpolicy::base_with.TcpServer,free_port, and a loopbackexchangeclient.Sandbox::run_python,ExecOutput::last_json,Sandbox::policy_revisions/active_policy,SandboxBuilder::providersandpolicy::binaries_yaml, plushost_server_support::TEST_HOST. Conformance no longer has asupportmodule.sandbox_clients_support::{connect, http, http_raw}calls. Credential tests read the provider placeholder withprintenvand send it withhttp.tests/support/sandbox-clients/scripts: the GraphQL, JSON-RPC, JSON-RPC audit, MCP, localhost, listener lifetime, preconnect owner, socket inheritance, reload keep-alive, echo server, WebSocket text and binary, and bound credential (bash) clients, exposed throughsandbox_clients_support::scripts.routable-host-ipno longer includes a conformance script from another workspace.network: bypass denial, reserved destinations, localhost without proxy env, listener lifetime.network_l7+network_l7/mcp: REST, GraphQL, JSON-RPC, and MCP.network_egress: reload, ambiguous update,tls: skip, middleware, pipelining.policy_updateandport_forward.credential_gatingadditions: header and body rewrite, WebSocket text rewrite, and static credentials bound to profile endpoints.tests/suites/features/routable-host-ip: the IP-literal endpoint test (explicitallowed_ipsand implicit). Policy DNS gives the sandbox a synthetic address forhost.openshell.internal, so the test reaches a host server by name first and uses the address the request arrived at (host-serverRequest::local_addr). CI rows: Docker rootful (deb), Podman rootful (rpm),k3s, andk3s-ha.policy_denied, and REST upgrades and the fail-closed middlewareunsupported_l7_protocol, as documented under "Proxy Error Responses".OPENSHELL_TEST_GATEWAY_REPLICAS=3, and the two tests that read logs (profile_endpoints_bind_static_credentials,fail_closed_middleware_blocks_uninspectable_traffic) skip themselves there. test(conformance): run the two sandbox-log tests on k3s-ha again #5752 tracks removing this.live_policy_updateis removed frome2e:rust:push. That lane already runsopenshell-conformanceagainst a push-mode gateway, so it runs thepolicy_updatetests instead. The tmachineconfig-pushenvironment is left to test(conformance): run the conformance suite against a gateway that pushes configuration #5622.static_provider_credentials_are_bound_to_profile_endpointsand its helpers fromhost_gateway_alias.rs. The other two tests in that file stay until PR5.[[test]]entries,PODMAN_CI_TESTSentries, andartifacts.nixfollow-up binary.e2e:websocket-conformancetask and the now-unusedSandboxGuard::spawn_forward.TESTING.md(routable-host-ip), and a note onContext::namethat sandbox suffixes can have at most 5 characters (test(testkit): share sandbox client helpers across testkit suites #5691'sMAX_SANDBOX_NAME_LENcheck enforces it).Defaults applied
host.openshell.internal. The legacy private-rangeallowed_ipsentries are gone.Sandbox::execinsleep infinitysandboxes, using pipes. The legacy tests were also non-TTY, so PTY/pipe parity is unchanged.exec_untilhelper;Sandbox::poll_execalready covers it.policy::base(),python_image(), and the shared Python binaries list.live_policy_updateround-trip and from-empty tests are merged intopolicy_set_versions_and_ignores_repeats.Deviations from the plan
port_forwardis its own module, not part ofcli.rs, to keep it separate from the CLI tests. It startsforward start -dwithcli_spawnand waits for the exit code, because the backgroundedsshkeeps the CLI's pipes open. A cleanup hook runsforward stop.credential_gating.rs. The unbound endpoint is a second port on the test host instead ofhost.docker.internal, so the 403 holds on every driver. The legacy Podmanconnect-deniedspecial case is gone.Rows that kept a container fixture
None. The IP-literal test needs a host address that the supervisor allows as an IP literal. Rootless Podman's address is link-local, which is always blocked. So instead of keeping a container fixture, that test is a feature package that does not run on rootless Podman.
Driver coverage changes
bypass_detection,forward_proxy_*,host_gateway_alias,live_policy_update, andno_proxy. It now runs all of the ported conformance tests, on rootful and rootless.host_gateway_aliasfrom this set. It now runs all of them throughopenshell-conformanceine2e-vm.sh. That is new coverage. If a newly covered row fails, I'll raise it here before skipping anything.Testing
cargo fmt --checkandcargo clippy --locked --all-targets -D warningsfortests/testkit,tests/support,tests/suites/conformance,tests/suites/featurescargo testfortests/testkitandtests/support(new host-server unit tests included)tasks/scripts/check-conformance-program.sh(77 tests)cargo check --testsfore2e/rustwith docker, podman, kubernetes, vm, and config-push features.cargo clippyonhost_gateway_aliasis clean.Checklist