Skip to content

test(e2e): port provider tests from e2e/rust to testkit - #5657

Draft
matthewgrossman wants to merge 11 commits into
mainfrom
test/5656-port-provider-tests/matthewgrossman
Draft

matthewgrossman wants to merge 11 commits into
mainfrom
test/5656-port-provider-tests/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Port the provider tests from e2e/rust to testkit and delete the legacy files. The provider files and auto-create tests join the conformance suite. The service bearer passthrough test moves to a new features/service-exposure suite.

Related Issue

Closes #5656
Part of #5281

Stack

Targets main, which now includes #5691. The provider files test attaches its provider with #5691's SandboxBuilder::providers(..), and the auto-create sandbox runs sleep infinity. The other #5691 helpers don't apply: the service-exposure server is the sandbox's main process, which sandbox-clients probes can't be, and no test runs Python scripts, names the host gateway, or writes policy YAML by hand.

Changes

  • testkit: create_provider and install_provider return a ProviderBuilder. Chain .config(key, value) to pass --config; existing calls are unchanged.
  • testkit: cx.service_get(url, headers) requests an exposed sandbox service through the gateway endpoint, with the service URL's authority as Host. It uses the gateway's mTLS bundle when the gateway has one, so tmachine environments need no extra routing.
  • testkit: testkit::Profile and testkit::Credential build provider profiles in Rust instead of format! YAML. Setters are named after the profile YAML fields, endpoints use policy::Endpoint (which gains enforce() and path()), and profile.to_yaml() renders through openshell-providers, the CLI's export path. Unit tests compare built profiles with the YAML they name. providers.rs and features/keycloak-provider-refresh use it; credential_gating.rs is converted in test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665.
  • conformance: providers.rs ports provider_file_open_update_and_detach and auto_created_provider_credential_available_in_sandbox.
  • features: new service-exposure package with bearer_passthrough_preserves_authorization_header, plus rows in .github/workflows/e2e-suites.yml.
  • Deleted provider_files.rs, provider_auto_create.rs, service_bearer_passthrough.rs, their [[test]] entries, the provider_auto_create entry in tests/tmachine/artifacts.nix, and the now-unused rustls and tokio-rustls e2e dependencies.

Differences from the legacy tests

  • Wider coverage. The provider tests run on every conformance row. Legacy provider_files ran on Docker only, and tmachine lanes skipped provider_auto_create. Service exposure runs on deb docker, rpm podman (rootful and rootless), k3s, and k3s-ha. Legacy ran it on Docker only.
  • Auto-create uses a run-unique profile. The legacy test relied on the harness importing every providers/*.yaml globally and used claude-code. The port imports its own run-unique profile and passes the credential through the CLI environment. That removes the global import that deferred this move in test(conformance): migrate workspace, upload, and template e2e tests #4246.
  • No provider-delete cleanup poll. The issue planned one. testkit already deletes sandboxes before providers, so the test doesn't need it.
  • The provider files test keeps the legacy path, /run/openshell/providers/<provider>/client.toml.

Testing

  • cargo fmt --check and cargo clippy -D warnings for testkit, conformance, features, and e2e/rust
  • testkit unit tests, including new service tests
  • E2E tests added/updated
  • E2E: every conformance row, all five features/service-exposure rows, all three features/keycloak-provider-refresh rows, and the legacy lanes pass in Branch E2E Checks. The provider conformance tests also pass locally against a Docker gateway.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b70615f

@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for b70615f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test c7af9e2

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b9d5b3e

matthewgrossman added a commit that referenced this pull request Nov 18, 2026
Signed-off-by: Matthew Grossman <[email protected]>

#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
matthewgrossman added a commit that referenced this pull request Nov 18, 2026
Signed-off-by: Matthew Grossman <[email protected]>

#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
@matthewgrossman
matthewgrossman changed the base branch from main to test/5689-share-sandbox-helpers/matthewgrossman November 18, 2026 18:26
create_provider and install_provider return a builder that creates the
provider when awaited, so tests can chain .config(key, value) for
provider create --config. Existing callers are unchanged.

Signed-off-by: Matthew Grossman <[email protected]>
Move provider_files and provider_auto_create from e2e/rust into a
providers conformance suite. The auto-create test imports a run-unique
profile instead of relying on a globally imported claude-code profile,
so it no longer changes shared gateway state.

Signed-off-by: Matthew Grossman <[email protected]>
Requests go to the gateway's endpoint with the service URL's host as the
Host header, which is how the gateway routes service traffic. Over TLS
the client trusts the gateway CA and presents the registered gateway's
client certificate, through openshell-crypto's rustls configuration.

Signed-off-by: Matthew Grossman <[email protected]>
@matthewgrossman
matthewgrossman force-pushed the test/5656-port-provider-tests/matthewgrossman branch from b9d5b3e to b5ef866 Compare November 18, 2026 18:37
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b5ef866

@matthewgrossman
matthewgrossman changed the base branch from test/5689-share-sandbox-helpers/matthewgrossman to main November 18, 2026 18:38
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test d79cc8a

laik pushed a commit to laik/OpenShell that referenced this pull request Nov 19, 2026
…IDIA#5691)

* test(testkit): add providers, run_python, last_json, policy readers and binaries_yaml

Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML.

Signed-off-by: Matthew Grossman <[email protected]>

* test(host-server): add TEST_HOST

Signed-off-by: Matthew Grossman <[email protected]>

* test(support): add sandbox-clients crate with connect and http probes

A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read.

Signed-off-by: Matthew Grossman <[email protected]>

* test(features): use shared helpers in host-gateway

The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process.

Signed-off-by: Matthew Grossman <[email protected]>

* test(suites): drop ready_sandbox wrappers and unify long-running commands

Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains.

Signed-off-by: Matthew Grossman <[email protected]>

* docs(testkit): list the shared sandbox helpers

Signed-off-by: Matthew Grossman <[email protected]>

* test(suites): use shared sandbox and provider helpers in cli, cdi and keycloak tests

The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand.

Signed-off-by: Matthew Grossman <[email protected]>

* test: trim unit tests for the shared sandbox helpers

The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet.

Signed-off-by: Matthew Grossman <[email protected]>

* test(features): leave the keycloak refresh test to NVIDIA#5657

Signed-off-by: Matthew Grossman <[email protected]>

NVIDIA#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.

* test(testkit): fail early on sandbox names the gateway rejects

The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit.

Signed-off-by: Matthew Grossman <[email protected]>

* docs(testkit): document how to write a sandbox client

The testkit README now explains why clients run in the sandbox, the contract a client script follows, and when a script moves to sandbox-clients. tcp_probe.py points to it, sandbox-clients says it holds network clients only, TESTING.md points Rust tests to run_python, and run_python suggests python_image() when the sandbox has no python3.

Signed-off-by: Matthew Grossman <[email protected]>

* test(conformance): use the shared TEST_HOST in credential gating

Signed-off-by: Matthew Grossman <[email protected]>

* fix(support): report DNS failures from the connect probe as dns, not errno

Signed-off-by: Matthew Grossman <[email protected]>

* refactor(support): drop the unused is_dns_failure helper

Signed-off-by: Matthew Grossman <[email protected]>

* refactor(support): parse HTTP replies with httparse

Signed-off-by: Matthew Grossman <[email protected]>

* fix(support): report a stalled send from the connect probe as a timeout

Signed-off-by: Matthew Grossman <[email protected]>

* docs(testkit): link the sandbox client guide and the server name limit

Signed-off-by: Matthew Grossman <[email protected]>

---------

Signed-off-by: Matthew Grossman <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(e2e): port provider tests from e2e/rust to testkit

1 participant