Skip to content

feat(testkit): run conformance as one program and add unsupported-test entries - #5755

Open
matthewgrossman wants to merge 4 commits into
mainfrom
feat/testkit-unsupported-tests/matthewgrossman
Open

matthewgrossman wants to merge 4 commits into
mainfrom
feat/testkit-unsupported-tests/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Make the openshell-conformance program the conformance test target, so run, cargo test and nextest all run the same code. Add OPENSHELL_TEST_UNSUPPORTED so an environment can mark a test unsupported, with the same skip under every way of running.

Related Issue

Closes #5754 (part of #5281)

Why an environment variable

The list is an environment variable and not a --skip flag:

  • --skip already means something else in two of the three ways of running tests. libtest's --skip PATTERN leaves out tests whose names match, and cargo nextest run accepts the same flag after --. So openshell-conformance --skip cli::x=https://... or cargo test -- --skip ... would be read as a name filter. It matches no test, so the test runs anyway, with no error. That is a silent misread, not just a confusing name.
  • A flag can't reach the program under nextest. nextest writes the program's command line itself (--list, then <name> --exact for each test). An environment variable is the only input every runner passes to every test process.
  • So one uniquely named variable is the one interface that behaves the same under openshell-conformance run, cargo test and nextest, which is the point of running one program everywhere.
  • "Unsupported" describes the environment ("this doesn't work here yet, see the issue"), not an action like a filter.
  • A later run --unsupported shortcut would be safe, because libtest rejects flags it doesn't know, so it could not be misread the way --skip would. Leaving it out keeps exactly one way to give the list.

Known limitation

Scoping by crate is what lets one variable serve several programs, but it means a misspelled crate prefix (conformence::smoke::x=...) is silently ignored. A program alone can't tell a typo from an entry meant for another program. The CI runner knows which programs a row runs, so it can reject entries for crates it does not run; that belongs with the CI step.

Follow-up wiring

Nothing sets OPENSHELL_TEST_UNSUPPORTED in CI yet; that is the CI step under #5281. It needs:

  • Matrix rows, host lane scripts and the VM lane to set the variable where an environment can't pass a test. For example, the on-failure restart test (bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507, a VM driver bug that hangs, so a skip fits) is still in e2e/rust; porting it to conformance and setting an entry for the VM lane would cover it.
  • The runner to carry the variable into the guest. tests/runner/src/test_env.rs already exports OPENSHELL_TEST_*=value lines from /etc/openshell-test/env.d/*.env, so an environment could publish it there. Whether a comma-separated value survives that path is unchecked.
  • The runner to reject entries for crates it does not run (see Known limitation).

Changes

  • harness = false on the openshell-conformance bin, test = false on the conformance library. nextest's binary ID changes from conformance to conformance::bin/openshell-conformance, and editor per-test "Run" buttons go away. The macro's #[tokio::test] wrappers are unchanged.
  • OPENSHELL_TEST_UNSUPPORTED: <test>=<https issue url> entries separated by commas and/or whitespace. Entries are qualified by crate, and a program applies only entries for crates it links.
  • Strict checks exit 2 before anything is listed or run: unknown, duplicate or #[ignore]d test, non-https:// issue, malformed entry.
  • Unsupported tests are ignored in libtest listing and runs, so every runner shows SKIP. list marks them, run prints the active entries once and lists them in its summary; under cargo test and nextest the program stays quiet and the caller lists the entries, and its JUnit marks them skipped with an unsupported property. run passes the variable to its test processes.
  • Forcing an unsupported test to run fails with unsupported: <issue>; remove the entry from OPENSHELL_TEST_UNSUPPORTED to run it.
  • Both READMEs document the variable.

Testing

  • Checks appropriate to the affected code and behavior pass (testkit and conformance fmt and clippy; testkit tests)
  • Unit tests added/updated (parsing, scoping, each check, JUnit, and program-level checks for each interface)
  • E2E tests added/updated (if applicable)

Also checked by hand on conformance without a gateway: cargo test, cargo nextest run and openshell-conformance run show the same skip for an entry, a forced run fails under each, and a bad entry exits 2 (nextest reports its own list-failure code, 104, which its listing step uses for any failure).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…t entries

Make the openshell-conformance bin the conformance test target
(harness = false) so run, cargo test and nextest execute the same
program. Add OPENSHELL_TEST_UNSUPPORTED, a list of <test>=<issue url>
entries that every interface reports as ignored, with startup,
summary and JUnit reporting, strict checks, and a refusal on forced
runs.

Signed-off-by: Matthew Grossman <[email protected]>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(testkit): run conformance as one program and skip unsupported tests per environment

1 participant