Skip to content

feat(review): automatically label accepted work - #5414

Draft
purp wants to merge 10 commits into
NVIDIA:mainfrom
purp:feat/5411-labeling-runner/purp
Draft

purp wants to merge 10 commits into
NVIDIA:mainfrom
purp:feat/5411-labeling-runner/purp

Conversation

@purp

@purp purp commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Connect the shared labeling helper to the managed classifier through one issue/PR runner. A number or URL produces a validated preview; explicitly authorized application reuses the saved proposal and exact preview, rechecks current evidence and eligibility, and preserves the helper's checkpoint and partial-failure recovery.

Related Issue

Closes #5411.

Based on #5407 (f2f0fae3a876d27e9848568baef89cff7630533e), with #5355's shared-helper branch (af08bc89249083f16dcb56b43ed56d1a6446a7e1) merged as a dependency. The parent branches live in the fork, so this draft targets main. Review the runner changes alone.

Changes

  • Add a preview-first one-item entry point with saved-preview --apply, human --force, and credential-free --request-preview; consolidate the temporary adapter CLI.
  • Stop deterministic preparation outcomes before model invocation. Support bounded evidence-expansion rounds using trusted checkout instructions and the current shared result contract, with an invocation-wide budget and deadline.
  • Delegate eligibility, classification validation, exact mutations, checkpoint verification and local ambiguity formatting to the shared helper. Persist session IDs before polling and retain execution references and uncertain outcomes without automatically replaying writes.
  • Document setup, concise reports, recovery and live prerequisites; register offline tests and code ownership. Automatic dispatch and coordination across entry points remain follow-up work.

Release Notes

Testing

  • 245 labeling tests pass: uv run --no-project --python /usr/bin/python3 --offline python -m unittest discover -s tasks/scripts -p '*label*test.py'. These include actual-helper orchestration with fake GitHub/CMA, deterministic skips, expansion, exact preview application, stale eligibility/content, verified checkpoints, uncertain creation/writes and recovery.
  • 28 accepted-work and 43 review-readiness Python tests pass; 19 JavaScript issue-acceptance tests pass.
  • Read-only preparation of live feat(review): connect the end-to-end labeling runner #5411 returns ready, classifier_required: true, with accepted/eligible work. No live CMA sessions or label writes were performed.
  • Task TOML, Python syntax, skill metadata/inventories, local links/anchors, crate inventory, new-file SPDX headers and staged diff checks pass.
  • Ruff, Markdownlint and the mise wrapper remain unverified: required tool downloads were blocked by network restrictions. Direct stdlib test commands above passed. Actionlint and external skill discovery were not run.
  • Observed live CMA configuration, previews and authorized one-item applications remain pending. This draft does not claim completion of that live-validation criterion.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Internal integration documentation and labeling skill updated

@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@purp purp changed the title feat(review): connect the one-item labeling runner feat(review): automatically label accepted work Nov 15, 2026
@purp purp added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:dx Making OpenShell easy to contribute to for humans and agents. labels Nov 15, 2026
purp added 8 commits November 17, 2026 16:48
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
The README links to NVIDIA#5276 for activation prerequisites and to the
modules for completion and invocation behavior instead of restating
them. Budget and timeout defaults live in one constant each and
appear in --help. A shared helper validates item numbers, and one
discover command runs both test files.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Apply ruff format, fix RUF021, B023, and RUF005, and add SPDX headers
so the scripts pass python:format:check, python:lint, and
license:check.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
@purp

purp commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Handoff: labeler layout and direct SDK integration

#5407 now contains the labeler layout migration. Rebase/integrate that change before resolving this branch's overlapping files. The convention is documented in .agents/agents/README.md: agent-root run.*, supporting code/tools in bin/, tests in tests/, descriptive-noun agent names.

Canonical directory: .agents/agents/labeler/. Module moves from tasks/scripts/:

Old module New module
managed_label_classifier.py bin/classifier.py
labeling_agent_events.py bin/events.py
label_accepted_work.py bin/acceptance.py
label_issue.py bin/reconciliation.py
label_issue_evidence.py bin/evidence.py
label_issue_policy.py bin/policy.py
label_issue_checkpoint.py bin/checkpoint.py
label_issue_github.py bin/github.py

Matching tests live in tests/<new-module>_test.py; the README is at the labeler root.

Adjust this PR:

  • Put label_issue_runner.py at the labeler root as run.py, and its tests at tests/run_test.py. Keep other supporting code in bin/; update imports, task commands, sparse checkouts, ownership, and documentation. Reconcile this branch's accepted_work.py changes into bin/acceptance.py rather than restoring a second acceptance implementation.
  • The entire label-issue skill, including references, now lives at skills/label-issue/ inside the labeler. .agents/skills/label-issue is a discovery symlink to it, not a second directory to edit. Preserve the alias; carry skill/contract edits into the canonical files. Do not split out a renamed classification skill or duplicate its policy in a prompt.
  • Every invocation goes through the deterministic runner first. Decide acceptance using established policy, lifecycle and material-change checks before creating a model session. A missing acceptance label does not itself mean ineligible: the intended end-to-end wrapper can reflect deterministically established acceptance, then classify. The model does not decide which work deserves acceptance. Triage delegates labeling through the labeler; retain direct read-only policy checks for other workflows.
  • ManagedAgents.request no longer exists. classifier.create_client returns the supported SDK client, and classifier.classify takes that client and calls client.beta.sessions directly. Use SDK-default retries and SDK pagination; do not restore the old HTTP-shaped transport or max_retries=0. Preserve this PR's session-reference persistence, execution budget and completion validation while adapting to the SDK.
  • Failure cleanup sends user.interrupt, polls session state, then archives; keep the stateful regression that rejects archival while running. The SDK retry policy does not authorize blind replay of uncertain GitHub label mutations.
  • Test tasks now explicitly select files, with PYTHONPATH=.agents/agents/labeler/bin:tasks/scripts. Preserve explicit selection and avoid running acceptance/helper suites twice.

Validation on #5407: 319 affected tests pass (including reviewer-readiness and stale closure); Ruff and relocated CLI imports pass. Alias target and Markdown links are verified. The external skill-discovery CLI remains unverified because its npm package is not cached. No live labeling/CMA operations were performed. Please resolve the integration in this PR's own working session.

purp added 2 commits November 19, 2026 01:39
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
@purp
purp force-pushed the feat/5411-labeling-runner/purp branch from 6b89b1d to 8f7afbb Compare November 19, 2026 03:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:dx Making OpenShell easy to contribute to for humans and agents.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(review): connect the end-to-end labeling runner

1 participant