Add live workflow status and original checklist to repository research audit - #2162
Open
groupthinking with Copilot wants to merge 16 commits into
Open
groupthinking with Copilot wants to merge 16 commits into
groupthinking with Copilot wants to merge 16 commits into
Conversation
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
groupthinking
September 20, 2026 01:12
View session
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: groupthinking/EventRelay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
🔍 PR Validation
|
github-actions
Bot
added
documentation
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
groupthinking
approved these changes
Sep 20, 2026
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
groupthinking
approved these changes
Sep 20, 2026
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Live workflow health can be inaccurate, checklist coverage is overstated, and workflow edits contradict the declared scope.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (3)
What changed in this PR
Adds repository-research auditing with optional live GitHub Actions health and checklist coverage reporting.
Changes:
- Adds six evidence-backed research skills and an orchestration harness.
- Collects repository evidence, live workflow data, and checklist statuses.
- Adds validation tests and an out-of-scope reconciliation workflow adjustment.
| File | Description |
|---|---|
tests/unit/test_runtime_repository_skills.py |
Tests runtime skill registration and orchestration. |
tests/unit/test_repository_research_skills.py |
Validates skill documentation and lock metadata. |
tests/unit/test_repository_research_audit.py |
Tests audit inputs, checklist output, and execution. |
tests/unit/test_repository_reconciliation_workflow.py |
Tests REST-based branch protection inventory. |
tests/unit/test_agent_operating_harness.py |
Tests harness validation behavior. |
src/skills/repository_research_common.py |
Adds shared evidence validation helpers. |
src/skills/repository_research_audit.py |
Implements evidence, workflow-health, and checklist collection. |
src/skills/product_positioning_and_buyer_fit/main.py |
Implements product-fit synthesis. |
src/skills/product_positioning_and_buyer_fit/__init__.py |
Defines the product-fit package. |
src/skills/persistence_and_boundary_truth/main.py |
Implements persistence and boundary synthesis. |
src/skills/persistence_and_boundary_truth/__init__.py |
Defines the persistence skill package. |
src/skills/github_ops_and_workflow_health/main.py |
Implements workflow-health synthesis. |
src/skills/github_ops_and_workflow_health/__init__.py |
Defines the workflow-health package. |
src/skills/engineering_risk_and_duplication_audit/main.py |
Implements risk-register synthesis. |
src/skills/engineering_risk_and_duplication_audit/__init__.py |
Defines the risk-audit package. |
src/skills/canonical_architecture_truth/main.py |
Implements architecture-truth synthesis. |
src/skills/canonical_architecture_truth/__init__.py |
Defines the architecture skill package. |
src/skills/agent_operating_harness/main.py |
Aggregates research skill outputs. |
src/skills/agent_operating_harness/__init__.py |
Defines the harness package. |
src/agents/mcp_ecosystem_coordinator.py |
Registers and executes repository research skills. |
skills-lock.json |
Registers the six new runtime skills. |
scripts/testing/run_repository_research_audit.py |
Adds the audit CLI runner. |
scripts/testing/agent_operating_harness.py |
Adds skill-bundle validation. |
.github/workflows/repository-reconciliation.yml |
Moves branch protection lookup to REST. |
.claude/skills/product-positioning-and-buyer-fit/SKILL.md |
Documents product-fit research guidance. |
.claude/skills/persistence-and-boundary-truth/SKILL.md |
Documents persistence research guidance. |
.claude/skills/github-ops-and-workflow-health/SKILL.md |
Documents workflow-health guidance. |
.claude/skills/engineering-risk-and-duplication-audit/SKILL.md |
Documents risk-audit guidance. |
.claude/skills/canonical-architecture-truth/SKILL.md |
Documents architecture research guidance. |
.claude/skills/agent-operating-harness/SKILL.md |
Documents harness ordering and outputs. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+150
to
+154
| { | ||
| (run.get("path") or run.get("name") or "<unknown>") | ||
| for run in runs | ||
| if run.get("conclusion") not in {None, "success", "skipped"} | ||
| } |
groupthinking
approved these changes
Sep 20, 2026
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
…an arbitrary global window of the 10 most recent workflow runs, causing false positives (stale failures) and false negatives (busy workflows dominating the window). This commit fixes the issue reported at src/skills/repository_research_audit.py:149 ## Bug In `collect_live_workflow_health` (src/skills/repository_research_audit.py), workflow health was computed from a single global endpoint: ```python runs_url = f"https://api.github.com/repos/{repo_slug}/actions/runs?per_page=10" ... failing_workflows = sorted({ (run.get("path") or run.get("name") or "<unknown>") for run in runs if run.get("conclusion") not in {None, "success", "skipped"} }) ``` `.../actions/runs?per_page=10` returns the 10 most recent runs **across all workflows**, ordered by recency. A workflow is flagged as failing if *any* run in that window failed. Two concrete failure modes: 1. **False positive (stale failure):** Workflow `A` failed at 10:00 then succeeded at 11:00. Both runs are in the window, so `A` is reported as failing even though its latest run passed. 2. **False negative / incompleteness:** If one busy workflow produces all 10 recent runs, every other workflow is absent from the window and its status is never evaluated — even a currently-failing workflow is silently omitted. `failing_workflows` is returned in the health report and interpolated into the item-18 checklist handling string in `_build_original_checklist`, so the audit surfaces misleading workflow status. ## Fix Compute health **per workflow** using each workflow's latest run. Iterate over the workflows from the workflows endpoint and query `.../actions/workflows/{id}/runs?per_page=1` for each, classifying that single latest run's `conclusion`. `recent_runs` and `failing_workflows` are now built from this deduplicated set of latest-per-workflow runs. Behavior preserved: * Fail-closed / graceful-degradation paths (no slug, no token, API error) are unchanged; the per-workflow requests are inside the same `try` block so any error still returns `status: "unavailable"`. * The returned dict shape (`recent_runs`, `failing_workflows`, `workflow_count`, etc.) is unchanged, keeping `_build_original_checklist` and the tests (which inject `live_workflow_health` directly) compatible. * The `{None, "success", "skipped"}` exclusion set is retained, so in-progress runs (conclusion `None`) are not counted as failing. Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com> Co-authored-by: groupthinking <garveyht@gmail.com>
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
groupthinking
approved these changes
Sep 20, 2026
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
Contributor
✅ E2E Test Results: ALL TESTS PASSED
Test Output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canonical issue
Closes #2182
Outcome
The repository research audit can now distinguish repo-structure workflow analysis from live GitHub workflow status, and it reports the original session checklist with item-by-item handling state.
Scope
live_workflow_healthplumbing into thegithub-ops-and-workflow-healthskill inputoriginal_checklistoutput withhandled/partially_handled/still_neededstates524in response metadataRisk
src/skills/repository_research_audit.pyand its companion testsVerification
Local validation for
a6c407719c1e6f966e5e13808e7afbc7f99e01d9(Python 3.12.12):Result: 224 passed, with two existing unawaited-coroutine warnings. Both timeout-response tests also pass with a fixed UUID containing
524; the original assertion fails under that same reproduction. Ruff passes for the harness and its regression tests.git diff --checkpasses.The reconciliation workflow and its companion tests are restored exactly to base
276d738891361bb8a7118afe0785d5b11f3c91b9, removing the unrelated workflow behavior from this PR. The main CI run passed ona6c407719c1e6f966e5e13808e7afbc7f99e01d9, including Python 3.10, 3.11, and 3.12. The issue-closing check also passes.All checks are not green: updating this PR description triggers the base
pull_request_targetRepository Reconciliation workflow. Its report job fails withResource not accessible by integrationin the GraphQL branch inventory query. This pre-existing base-workflow defect is tracked by #2165 and requires a separately scoped fix onmain; changing this PR's workflow file cannot repair a run using the base workflow. Latest completed snapshot: 27 passed, three neutral/skipped, one failed reconciliation report.gh pr checks --requiredreports no configured required checks.The evidence-validation and scope review threads are resolved. The non-blocking workflow enabled/disabled/no-run reporting concern remains open.
Production evidence
Not applicable. This change affects repository audit/harness tooling, not the product runtime or deployment surface.
Agent handoff