Skip to content

feat(review): add labeling agent in Claude Managed Agents - #5407

Draft
purp wants to merge 10 commits into
NVIDIA:mainfrom
purp:feat/5276-labeling-agent-runtime/purp
Draft

purp wants to merge 10 commits into
NVIDIA:mainfrom
purp:feat/5276-labeling-agent-runtime/purp

Conversation

@purp

@purp purp commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a manually invocable labeling agent at .agents/agents/labeler/run.py: one issue/PR number or URL runs deterministic preparation, bounded CMA classification and evidence expansion, then returns a validated preview. Explicitly authorized application reuses the saved exact preview and shared reconciliation helper. Automatic triggering remains disabled.

Related Issue

Closes #5406. Starts Part 3 of #5276; this PR does not close the overall tracker.

Based on main after #5260 and #5355 merged; no temporary tracking branch is required.

Integration order: #5260 supplies accepted-work policy and opening-time PR acceptance labeling; #5355 supplies the standalone skill and migration helpers; #5151 supplies label/routing preparation before observed manual migration; automatic acceptance-triggered labeling follows those observed runs.

Changes

  • Combine the CMA adapter and one-item runner from feat(review): automatically label accepted work #5414. Pin the agent version and SDK; bound classification rounds, input size, event volume, budget and execution time. SDK retries can extend individual requests beyond the polling deadline.
  • Assemble trusted checkout instructions separately from untrusted evidence. Disable classifier tools, MCP servers and attached skills; supply no GitHub credentials to CMA.
  • Persist session IDs before polling and retain proposals, exact previews and recovery reports. On classifier failure or interruption, attempt interruption, wait for idle and archive; report unconfirmed cleanup.
  • Reuse the shared helper for eligibility, evidence expansion, semantic validation, fresh-state checks, label writes and verified checkpoints. Saved application does not reclassify.
  • Move labeler-owned helpers, skill and tests into the agent layout, with a linked skill-discovery alias. Update references, ownership and CI; discover the isolated test directory once.
  • Include deterministic acceptance-event and authorized human invocation selection as a library; install no automatic dispatch workflow.

Before activation: provision and verify the dedicated CMA configuration, observe representative previews and authorized one-item applications, and complete cross-host coordination and acceptance-writer dispatch in the follow-up. A GITHUB_TOKEN acceptance-label write needs an explicit dispatch hook, including drafts and independently of subsequent reviewer-routing success.

Release Notes

Testing

  • 349 offline tests pass using Python 3.12: 283 labeler tests, 61 review-readiness tests and 5 stale-closure tests. Run the commands registered in tasks/test.toml directly with their configured PYTHONPATH values and UV_PYTHON=/usr/bin/python3.
  • The root entry point's --help runs; git diff --check passes.
  • Runner tests exercise the actual shared helper with fake GitHub and SDK services, including deterministic skips, expansion, preview, saved application and recovery paths. No remote sessions or writes occur in these tests.
  • The integrated runner commit is signed, DCO-signed-off and verified by GitHub; signing preserves its code tree and message.
  • The mise wrapper and configured tooling remain unverified: required tool downloads are blocked. Ruff, Markdownlint and Actionlint were not run for this integration.
  • Live CMA configuration, representative previews and authorized applications remain unverified. No live CMA sessions or label mutations were performed in this integration pass.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off (DCO).
  • Internal integration documentation updated.

@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@purp purp changed the title feat(review): add managed labeling runtime foundation feat(review): automatically label issues and PRs using Claude Managed Agents Nov 9, 2026
@purp purp changed the title feat(review): automatically label issues and PRs using Claude Managed Agents feat(review): tools for labeling using Claude Managed Agents Nov 9, 2026
@purp purp changed the title feat(review): tools for labeling using Claude Managed Agents feat(review): tools for labeling agent Nov 10, 2026
@purp purp changed the title feat(review): tools for labeling agent feat(review): add labeling agent in Claude Managed Agents Nov 10, 2026
@purp
purp force-pushed the feat/5276-labeling-agent-runtime/purp branch from 64ff126 to 103e234 Compare November 11, 2026 18:56
@purp
purp changed the base branch from main to chore/5276-base-5355/purp November 11, 2026 18:56
@purp
purp force-pushed the chore/5276-base-5355/purp branch from 6c3bf15 to 29b4687 Compare November 11, 2026 21:58
@purp
purp force-pushed the feat/5276-labeling-agent-runtime/purp branch 2 times, most recently from f0ef3dd to b2ef00f Compare November 12, 2026 13:16
@purp
purp changed the base branch from chore/5276-base-5355/purp to main November 12, 2026 13:16
@purp purp added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:dx Making OpenShell easy to contribute to for humans and agents. labels Nov 12, 2026
@purp

purp commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Migration experiment: runtime follow-up

Observed Sonnet/Haiku comparison should inform this adapter and #5411's integration:

This was fixed-evidence, tool-free Claude CLI testing, not validation of CMA execution. No label writes occurred. #5413 owns reproducible quality/cost/latency evaluation.

@purp
purp force-pushed the feat/5276-labeling-agent-runtime/purp branch from b2ef00f to 3876a8a Compare November 14, 2026 23:52
@purp purp added meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. and removed meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. labels Nov 15, 2026
@purp

purp commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

The new CI step re-runs the existing label tests, and the PR body's test count is wrong.

test:labeling-agent discovers *label*_test.py (tasks/test.toml). That pattern also matches the six existing label_* test files under tasks/scripts/, so the new "Test labeling agent runtime" step in Branch Checks runs those suites a second time, alongside their own tasks such as test:accepted-work. As a result, the task does more than its description says, and the "30 tests" in the PR body undercounts what it runs.

Narrowing the pattern to the two new files (labeling_agent_events_test.py and managed_label_classifier_test.py) would fix both.

@purp purp added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
purp added 4 commits November 17, 2026 16:48
The README links to NVIDIA#5276 for activation prerequisites and to the
modules for completion and invocation behavior instead of restating
them. Budget and timeout defaults live in one constant each and
appear in --help. A shared helper validates item numbers, and one
discover command runs both test files.

Signed-off-by: Jim Meyer <[email protected]>
Apply ruff format, fix RUF021, B023, and RUF005, and add SPDX headers
so the scripts pass python:format:check, python:lint, and
license:check.

Signed-off-by: Jim Meyer <[email protected]>
@purp
purp force-pushed the feat/5276-labeling-agent-runtime/purp branch from 3876a8a to fba1880 Compare November 17, 2026 16:53
@purp

purp commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

gator-agent

Blocked

PR #5407 is still a draft, so Gator is waiting to begin code review.

Action required: @purp, mark the PR ready for review when it is ready for independent review.

Gator metadata
  • Head SHA: 459ca9dad4cb5f890aabd8a2c85eb05ec50178f3
  • Gator payload: 13
  • Next state: gator:blocked
  • Blocked reason: draft_pr

@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@purp
purp force-pushed the feat/5276-labeling-agent-runtime/purp branch from 3131b1e to 855c350 Compare November 19, 2026 04:13
@purp

purp commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

gator-agent

Blocked

PR #5407 remains a draft, so independent code review has not started.

Action required: @purp, mark the PR ready for review when it is ready for independent review.

Gator metadata
  • Head SHA: 855c350073ba56e123494d28643a70a87e443fde
  • Gator payload: 13
  • Next state: gator:blocked
  • Blocked reason: draft_pr

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. gator:blocked Gator is blocked by process or repository gates state:accepted A maintainer decided OpenShell should pursue this issue topic:dx Making OpenShell easy to contribute to for humans and agents.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(review): deliver managed labeling runtime and one-item runner

1 participant