Repository navigation
feat(testkit): run conformance as one program and add unsupported-test entries - #5755
Open
matthewgrossman wants to merge 4 commits into
Open
matthewgrossman wants to merge 4 commits into
matthewgrossman wants to merge 4 commits into
Conversation
…t entries Make the openshell-conformance bin the conformance test target (harness = false) so run, cargo test and nextest execute the same program. Add OPENSHELL_TEST_UNSUPPORTED, a list of <test>=<issue url> entries that every interface reports as ignored, with startup, summary and JUnit reporting, strict checks, and a refusal on forced runs. Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
…der libtest Signed-off-by: Matthew Grossman <[email protected]>
Signed-off-by: Matthew Grossman <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the
openshell-conformanceprogram the conformance test target, sorun,cargo testand nextest all run the same code. AddOPENSHELL_TEST_UNSUPPORTEDso an environment can mark a test unsupported, with the same skip under every way of running.Related Issue
Closes #5754 (part of #5281)
Why an environment variable
The list is an environment variable and not a
--skipflag:--skipalready means something else in two of the three ways of running tests. libtest's--skip PATTERNleaves out tests whose names match, andcargo nextest runaccepts the same flag after--. Soopenshell-conformance --skip cli::x=https://...orcargo test -- --skip ...would be read as a name filter. It matches no test, so the test runs anyway, with no error. That is a silent misread, not just a confusing name.--list, then<name> --exactfor each test). An environment variable is the only input every runner passes to every test process.openshell-conformance run,cargo testand nextest, which is the point of running one program everywhere.run --unsupportedshortcut would be safe, because libtest rejects flags it doesn't know, so it could not be misread the way--skipwould. Leaving it out keeps exactly one way to give the list.Known limitation
Scoping by crate is what lets one variable serve several programs, but it means a misspelled crate prefix (
conformence::smoke::x=...) is silently ignored. A program alone can't tell a typo from an entry meant for another program. The CI runner knows which programs a row runs, so it can reject entries for crates it does not run; that belongs with the CI step.Follow-up wiring
Nothing sets
OPENSHELL_TEST_UNSUPPORTEDin CI yet; that is the CI step under #5281. It needs:e2e/rust; porting it to conformance and setting an entry for the VM lane would cover it.tests/runner/src/test_env.rsalready exportsOPENSHELL_TEST_*=valuelines from/etc/openshell-test/env.d/*.env, so an environment could publish it there. Whether a comma-separated value survives that path is unchecked.Changes
harness = falseon theopenshell-conformancebin,test = falseon the conformance library. nextest's binary ID changes fromconformancetoconformance::bin/openshell-conformance, and editor per-test "Run" buttons go away. The macro's#[tokio::test]wrappers are unchanged.OPENSHELL_TEST_UNSUPPORTED:<test>=<https issue url>entries separated by commas and/or whitespace. Entries are qualified by crate, and a program applies only entries for crates it links.#[ignore]d test, non-https://issue, malformed entry.listmarks them,runprints the active entries once and lists them in its summary; undercargo testand nextest the program stays quiet and the caller lists the entries, and its JUnit marks themskippedwith anunsupportedproperty.runpasses the variable to its test processes.unsupported: <issue>; remove the entry from OPENSHELL_TEST_UNSUPPORTED to run it.Testing
Also checked by hand on conformance without a gateway:
cargo test,cargo nextest runandopenshell-conformance runshow the same skip for an entry, a forced run fails under each, and a bad entry exits 2 (nextest reports its own list-failure code, 104, which its listing step uses for any failure).Checklist