Skip to content

test(testkit): share sandbox client helpers across testkit suites - #5691

Merged
matthewgrossman merged 17 commits into
mainfrom
test/5689-share-sandbox-helpers/matthewgrossman
Nov 19, 2026
Merged

matthewgrossman merged 17 commits into
mainfrom
test/5689-share-sandbox-helpers/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds shared helpers for tests that run clients inside a sandbox, so later network, policy and credential suites don't each write their own. Converts the existing copies on this branch to use them. #5665 (PR2) and #5695 (PR5, issue #5676) stack on this. #5657 (PR3) is on main. #5624 is on main and adopts the helpers after this merges.

Related Issue

Closes #5689
Part of #5281

Stack: main ← #5691 ← #5665 ← #5695. #5442 has merged, and this branch is rebased onto main.

Changes

  • testkit: SandboxBuilder::providers(..), Sandbox::run_python(script, args) (fails on a nonzero exit), ExecOutput::last_json() (last JSON object or array line of stdout), Sandbox::policy_revisions() / active_policy(), and policy::binaries_yaml(..) for hand-written policy YAML.
  • Sandbox name check: cx.sandbox, cx.cli_sandbox and cx.track panic at the test's call when a name exceeds the gateway's 19 characters (testkit::MAX_SANDBOX_NAME_LEN), which leaves 5 for a suffix. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 found that 21 of its 22 E2E failures came from long names. Context::name is unchanged, because provider and profile names have no limit, so test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 doesn't need to change it.
  • host-server: TEST_HOST (host.openshell.internal).
  • New tests/support/sandbox-clients crate (sandbox_clients_support): one Python probe script, plus connect, http and http_raw, which run it and return typed results (connect outcome with errno, or a separate DNS-failure flag when the name doesn't resolve; how the read ended, received bytes, parsed HTTP response). HTTP is parsed in Rust. http sends Connection: close; http_raw stops reading after 2 idle seconds, so a keep-alive server can't stall a test.
  • Docs: the testkit README has a "Writing a sandbox client" section: why the client runs in the sandbox, the script contract (standard library only, input as arguments, one JSON line, failures as results, parsing in Rust), and when a script beside a test moves to sandbox-clients. tcp_probe.py and the crate docs point to it, sandbox-clients says it holds network clients only, TESTING.md points Rust tests to run_python, and run_python suggests python_image() when the sandbox has no python3.
  • No long-running sandbox shorthand. Tests write .image(testkit::python_image()).command(["sleep", "infinity"]); the testkit README shows this as the standard pattern.
  • Conversions:
  • credential_gating.rs uses the shared TEST_HOST; its clients take the host and port as arguments. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 rewrites the rest of that file.

Testing

  • cargo fmt --check and cargo clippy --locked --all-targets -- -D warnings for tests/testkit, tests/support, tests/suites/conformance, tests/suites/features and tests/suites/drivers/podman
  • cargo nextest run for tests/testkit and tests/support. Unit tests for the new helpers are limited to two HTTP parser cases no e2e test reaches yet; the e2e suites cover the rest
  • Cargo lockfile checks
  • Probe script checked locally against real sockets: clean close, keep-alive (idle and timeout), reset, refused and connect-only
  • Local Docker gateway: full conformance suite (43/43) and features/host-gateway pass. CDI and Keycloak need a GPU or Keycloak and run in CI
  • connect-ssh-supervision passes clippy in a Linux container
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 18, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 70aaaf3

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test dcf9567

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 70adbdc

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 4fa87f9

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 18, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 4fa87f9. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b76ff7b

@matthewgrossman
matthewgrossman marked this pull request as ready for review November 18, 2026 06:26
Base automatically changed from test/5426-port-cli-attach-exec/matthewgrossman to main November 18, 2026 09:50

@elezar elezar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design looks sound.

  • The client runs inside the sandbox, so policy applies to it.
  • The probe script only moves bytes and prints one JSON line, and HTTP parsing is unit-tested in Rust.
  • The script is embedded in the test binary, so it can't drift from its parser.
  • Failures include stderr in testkit reports.

I read the helper code and docs and skimmed the converted suites. I did not build or run anything.

To address

  1. Document the sandbox-script pattern in the testkit README. It lists the helpers but not why the client runs in the sandbox, the script contract (one JSON line, failures as results, argv input, stdlib only, parsing in Rust), or where a new script goes. Please also say that one-off scripts may stay beside a test and when to promote one to sandbox-clients. Many legacy e2e/rust/tests/ files carry hand-rolled clients of this shape (bypass_detection.rs, forward_proxy_*_l7.rs), and DNS, TLS and listener probes will follow.
  2. #5442 is merged, but the diff against main still shows about 40 commits and 2.4k added lines. Could you rebase so this PR shows only the helper change?
  3. Closes #5689 isn't registered as a closing reference (closingIssuesReferences is empty), and #5689 doesn't have state:accepted.

Suggestions

  • A hint toward testkit::python_image() when run_python fails with python3: not found.
  • TESTING.md mentions Python only for the pytest-era exec_python. A pointer to run_python would avoid confusion.

For #5665: when it rewrites credential_gating.rs, it can drop the local run_python for Sandbox::run_python. BODY_CLIENT may fit the shared http helper if it can send a body and headers (not checked). WEBSOCKET_CLIENT is a candidate for a script file.

Comment thread tests/testkit/testkit/src/context.rs Resolved
Comment thread tests/testkit/README.md Resolved
…nd binaries_yaml

Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML.

Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read.

Signed-off-by: Matthew Grossman <[email protected]>
The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process.

Signed-off-by: Matthew Grossman <[email protected]>
…ands

Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains.

Signed-off-by: Matthew Grossman <[email protected]>
… keycloak tests

The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand.

Signed-off-by: Matthew Grossman <[email protected]>
The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet.

Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>

#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit.

Signed-off-by: Matthew Grossman <[email protected]>
The testkit README now explains why clients run in the sandbox, the contract a client script follows, and when a script moves to sandbox-clients. tcp_probe.py points to it, sandbox-clients says it holds network clients only, TESTING.md points Rust tests to run_python, and run_python suggests python_image() when the sandbox has no python3.

Signed-off-by: Matthew Grossman <[email protected]>
@matthewgrossman
matthewgrossman force-pushed the test/5689-share-sandbox-helpers/matthewgrossman branch from b76ff7b to 6eae90b Compare November 18, 2026 18:12
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 6eae90b

@matthewgrossman

Copy link
Copy Markdown
Member Author

Thanks @elezar. All addressed in 6eae90b:

  1. Sandbox-script docs: the testkit README has a new "Writing a sandbox client" section, and tcp_probe.py, the crate docs and TESTING.md point to it (details in the inline replies).
  2. Rebase: the branch is rebased onto main and shows only this PR's 11 commits. One change came out of it: fix(server): find a finished sandbox's supervisor on any replica #5667 made features/host-gateway run its client as the sandbox's main process, to cover attaching to a finished process through another replica. The probes run through exec, so host-gateway keeps its inline client and now only uses TEST_HOST. No test on this branch calls http(..) yet; test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 is the first.
  3. Closing reference: Closes #5689 is now registered, and test(testkit): share sandbox client helpers across testkit suites #5689 has state:accepted.

The two suggestions are in too: run_python hints at python_image() when python3 is missing, and TESTING.md points Rust tests to run_python.

For #5665: it already drops its local run_python for Sandbox::run_python, and its clients, including the WebSocket one, are shared scripts in sandbox_clients_support::scripts.

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 06b770c

@pimlock

pimlock commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

This looks good to ship. The shared probe and testkit helpers fit the stacked PRs well, and the sandbox name check will save a lot of confusing gateway errors.

A few nits to consider, none blocking:

  • MAX_SANDBOX_NAME_LEN mirrors the server's MAX_ROUTABLE_NAME_LEN (crates/openshell-server/src/grpc/mod.rs). Naming that constant in the doc comment would help keep the two in sync.
  • message.rs has a hand-written HTTP response parser. httparse is already in tests/support/Cargo.lock (via tungstenite) and could replace the status-line, header, and chunk-size parsing. Fine to keep the lenient parser if that is intentional for proxy tests.
  • ConnectOutcome::is_dns_failure and Response::header have no caller in this PR or in test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 / test(e2e): port Docker and Podman driver feature tests from e2e/rust #5695.
  • In tcp_probe.py, sendall runs while the socket still has the connect timeout, before the read deadline starts, so a stalled send is reported as failed rather than timeout.
  • A truncated Content-Length or chunked body is returned with no completeness flag, and non-UTF-8 header bytes make response None. Probably fine for now, but worth knowing when debugging.
  • In TESTING.md, "the README's" is a bit ambiguous; linking tests/testkit/README.md#writing-a-sandbox-client would be clearer.

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test e38f314

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test a81fb32

@matthewgrossman

Copy link
Copy Markdown
Member Author

Thanks for the review. All addressed:

  • Parser: the response head and chunk sizes now go through httparse (e38f314). Header values with invalid UTF-8 are now decoded leniently instead of dropping the response.
  • Unused helpers: I dropped is_dns_failure (9ecf6b7). Response::header stays because the parser uses it.
  • Stalled send: the probe now starts its read deadline before sending, so a stalled send reports timeout (38d122c).
  • Docs (a81fb32): MAX_SANDBOX_NAME_LEN names MAX_ROUTABLE_NAME_LEN. Response::body notes that a cut-short body is returned as far as it arrived and that read_end says why reading stopped. TESTING.md links the "Writing a sandbox client" section.
  • Completeness flag: I didn't add one. read_end plus the raw bytes already show a cut-short body, so I'd wait until a test needs it.

Comment thread TESTING.md Resolved
@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 2c3fc96 Nov 19, 2026
160 of 161 checks passed
@matthewgrossman
matthewgrossman deleted the test/5689-share-sandbox-helpers/matthewgrossman branch November 19, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(testkit): share sandbox client helpers across testkit suites

3 participants