feat(evaluators): quarry for controls

.github/workflows/ci.yml

@@ -8,6 +8,10 @@ on:
 jobs:
   ci:
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.12", "3.13", "3.14"]
     services:
       postgres:
         image: postgres:16-alpine
@@ -24,6 +28,8 @@ jobs:
           --health-retries=5
 
     env:
+      # Override .python-version so uv uses the interpreter selected by this job.
+      UV_PYTHON: ${{ matrix.python-version }}
       DB_HOST: 127.0.0.1
       DB_PORT: "5432"
       DB_USER: agent_control
@@ -37,7 +43,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.12"
+          python-version: ${{ matrix.python-version }}
 
       - name: Setup uv
         uses: astral-sh/setup-uv@v4
@@ -59,8 +65,25 @@ jobs:
         run: make sync
 
       - name: Run Python checks
+        if: matrix.python-version != '3.14'
         run: make check
 
+      - name: Build and smoke test Quarry on Python 3.13
+        if: matrix.python-version == '3.13'
+        run: make -C evaluators/contrib/quarry build example
+
+      - name: Run Python 3.14 checks (Quarry blocked by upstream metadata)
+        if: matrix.python-version == '3.14'
+        run: |
+          make check -o contrib-test -o contrib-lint -o contrib-typecheck
+          for package in $(uv run python scripts/contrib_packages.py names); do
+            if [ "$package" = "quarry" ]; then
+              echo "Quarry excluded: cisco-project-quarry==0.1.0 requires Python <3.14."
+              continue
+            fi
+            make -C "evaluators/contrib/$package" check
+          done
+
       - name: Collect coverage files
         id: coverage_files
         run: |

evaluators/builtin/pyproject.toml

@@ -19,6 +19,7 @@ galileo = ["agent-control-evaluator-galileo>=8.6.0"]
 budget = ["agent-control-evaluator-budget>=7.5.0"]
 cisco = ["agent-control-evaluator-cisco>=7.5.0"]
 defenseclaw = ["agent-control-evaluator-defenseclaw>=8.2.0"]
+quarry = ["agent-control-evaluator-quarry>=8.15.0; python_version < '3.14'"]
 dev = ["pytest>=8.0.0", "pytest-asyncio>=0.23.0"]
 
 [project.entry-points."agent_control.evaluators"]
@@ -41,3 +42,4 @@ agent-control-evaluator-galileo = { path = "../contrib/galileo", editable = true
 agent-control-evaluator-budget = { path = "../contrib/budget", editable = true }
 agent-control-evaluator-cisco = { path = "../contrib/cisco", editable = true }
 agent-control-evaluator-defenseclaw = { path = "../contrib/defenseclaw", editable = true }
+agent-control-evaluator-quarry = { path = "../contrib/quarry", editable = true }

evaluators/builtin/tests/test_contrib_packages.py

@@ -41,6 +41,7 @@ def test_discover_contrib_packages_returns_expected_metadata() -> None:
         ("cisco", "agent-control-evaluator-cisco", "cisco"),
         ("defenseclaw", "agent-control-evaluator-defenseclaw", "defenseclaw"),
         ("galileo", "agent-control-evaluator-galileo", "galileo"),
+        ("quarry", "agent-control-evaluator-quarry", "quarry"),
     ]
 
 

evaluators/contrib/README.md

@@ -3,6 +3,7 @@
 Contributed evaluators and templates for extending Agent Control.
 
 - `galileo/` — Galileo evaluator integrations (Luna, Decision Models, LLM-as-judge)
+- `quarry/` — Local Project Quarry policies over trace/session steps (Python 3.12–3.13)
 - `template/` — Starter template for adding new evaluators
 
 Full guide: https://docs.agentcontrol.dev/concepts/evaluators/custom-evaluators

evaluators/contrib/quarry/Makefile

@@ -0,0 +1,24 @@
+.PHONY: sync test lint lint-fix typecheck check build example
+
+sync:
+	uv sync
+
+test:
+	uv run pytest tests --cov=src --cov-report=xml:../../../coverage-evaluators-quarry.xml -q
+
+lint:
+	uv run ruff check --config ../../../pyproject.toml src/ tests/ examples/
+
+lint-fix:
+	uv run ruff check --config ../../../pyproject.toml --fix src/ tests/ examples/
+
+typecheck:
+	uv run mypy --config-file ../../../pyproject.toml src/
+
+check: test lint typecheck
+
+build:
+	uv build
+
+example:
+	uv run python examples/local_policy.py

evaluators/contrib/quarry/PYTHON314.md

@@ -0,0 +1,201 @@
+# Python 3.14 compatibility audit
+
+Audit date: November 9, 2026. Branch: `feature/quarry-for-controls`.
+Implementation baseline: `aff76246c9b003ab3553ea6fa32b5bbc880d0367`.
+
+## Supported deployment boundary
+
+Python **3.13** is the supported integration target for Orbit deployments that
+load the Quarry evaluator. Both `agent-control-evaluator-quarry==8.15.0` and its
+pinned upstream dependency `cisco-project-quarry==0.1.0` admit that interpreter.
+They can be installed with normal metadata resolution; no fork, metadata bypass,
+or upstream release is needed for this target. Python 3.12 is also admitted.
+
+Agent Control core continues supporting Python 3.14 **without Quarry**. A process
+loading Quarry must use a supported interpreter throughout its image, toolchain,
+and dependency environment. A source pin alone does not make an in-process
+Python 3.14 Quarry installation supported. Orbit's image/toolchain update belongs in
+Orbit; this repository supplies the compatible evaluator and CI coverage.
+
+CI now exercises 3.12, 3.13, and 3.14. The 3.13 job runs the complete `make check`
+including Quarry, engine, and builtin evaluator tests, then builds Quarry's
+wheel/sdist and runs the offline example. The 3.14 job continues checking core
+and other contrib packages with an explicit Quarry exclusion.
+
+## Python 3.13 normal installation validation
+
+On CPython 3.13.13, the following completed with normal metadata resolution and
+without `UV_NO_SYNC`, `--ignore-requires-python`, or dependency overrides:
+
+```sh
+UV_PYTHON=3.13 make -C evaluators/contrib/quarry sync check example build
+```
+
+All 54 Quarry tests passed, along with lint, typecheck, the real offline scan
+example, and wheel/sdist builds. The resulting 40-package development environment
+passed `uv pip check`; all installed distributions admit the interpreter.
+
+A clean source snapshot also passed the complete root `make sync check` under
+Python 3.13.13, without excluding any target: 2,928 tests passed and two SDK
+tests skipped because their optional Galileo extras were absent. This includes
+138 engine, 311 builtin evaluator, and 54 Quarry tests, plus all repository lint
+and typechecks. Each contrib package used its own normally resolved environment.
+
+A separate clean environment installed the built builtin evaluator and models
+wheels with `agent-control-evaluators[quarry]`, resolving Quarry from its built
+wheel. All 27 runtime distributions passed dependency and interpreter metadata
+checks. Entry-point discovery and an actual matching policy scan succeeded from
+installed `site-packages`, with no editable source imports.
+
+Built metadata matches the manifests: our Quarry wheel retains
+`Requires-Python: >=3.12,<3.14` and `cisco-project-quarry==0.1.0`. The builtin extra
+selects Quarry for Python 3.13 and excludes it for Python 3.14. This validates the
+supported Python 3.13 install path without changing the per-package boundary.
+
+The same normal `sync check example build` targets also passed in a native Linux
+ARM64 container on CPython 3.13.18: 54 tests, lint, typecheck, a matching scan, and
+wheel/sdist builds. All 40 installed development distributions passed dependency
+checks and an explicit check of their `Requires-Python` bounds. No metadata was
+bypassed. Orbit's actual image/Bazel integration remains a downstream check.
+
+## Python 3.14 Quarry integration remains blocked
+
+There is no supported Python 3.14 Quarry dependency to pin yet. The only published
+release, `cisco-project-quarry==0.1.0`, declares `Requires-Python: >=3.11,<3.14`.
+Upstream `main` also excludes 3.14. Sources:
+[release metadata](https://pypi.org/pypi/cisco-project-quarry/0.1.0/json),
+[upstream manifest](https://github.com/cisco-ai-defense/project-quarry/blob/main/pyproject.toml).
+
+Agent Control's Quarry package correctly retains `>=3.12,<3.14`, and the builtin
+`quarry` extra still excludes Python 3.14. Removing only those two restrictions
+would leave an incompatible upstream distribution. An upstream release or an
+approved immutable compatibility fork is needed before widening our metadata.
+Do not treat the baseline SHA as an Orbit-compatible Python 3.14 source pin.
+
+## Workspace metadata, builds, and automation
+
+| Area | Finding |
+| --- | --- |
+| Workspace root and server | `>=3.12,<3.15`; already admit 3.14. |
+| Models, engine, telemetry, builtin evaluators, Python SDK | `>=3.12`; already admit 3.14. |
+| Budget, Cisco, DefenseClaw, Galileo contrib packages | `>=3.12`; install and pass checks on 3.14. |
+| Quarry contrib | `>=3.12,<3.14`; normal `make sync` rejects a 3.14 interpreter. |
+| Standalone example manifests | All declare `>=3.12`; no additional metadata exclusion. Their separate framework/application environments were not all executed. |
+| Classifiers | The SDK was the only package listing individual Python versions; 3.14 was added after its tests passed. |
+| Interpreter selection | `.python-version` remains `3.12.13`. Checks explicitly set `UV_PYTHON=3.14`; changing `setup-python` alone would not override this file. |
+| CI | Matrix covers 3.12, 3.13, and 3.14. The 3.13 job checks and builds Quarry normally; 3.14 explicitly excludes Quarry. |
+| Release builds | Python publishing and the TypeScript release helper use 3.12. Their packaging does not itself forbid consumers on 3.14. The existing release defaults remain unchanged. |
+| Server Docker image | Uses `ghcr.io/astral-sh/uv:python3.12-bookworm-slim`; its default was not changed. Orbit's separate image and Bazel toolchain were not modified or validated here. |
+| Bazel | This repository has no Bazel configuration to update. |
+| Wheels/sdists | Root `make build` successfully built core and all contrib packages under 3.14. Building a universal wheel does not establish compatible `Requires-Python` metadata. |
+
+## Dependency closure
+
+The evaluator's runtime closure was traversed with active extras, including
+`sqlglot[c]`. All 23 third-party dependencies below install under CPython 3.14.7
+on macOS ARM64. Only our Quarry package and upstream Quarry fail the interpreter
+metadata check. These were scratch environments; no production requirement or
+upstream wheel metadata was rewritten.
+
+```text
+annotated-types==0.8.0       attrs==26.1.0
+cachetools==7.2.0           google-re2==1.1.20251105
+jsonschema==4.26.0          jsonschema-specifications==2025.9.1
+lark==1.3.1                loguru==0.7.3
+markdown-it-py==4.2.0       mdurl==0.1.2
+orjson==3.13.0             packaging==26.3
+pydantic==2.14.1           pydantic-core==2.50.1
+pygments==2.21.0           pyyaml==6.0.3
+referencing==0.37.0        rich==15.0.0
+rpds-py==2026.9.1          sqlglot==30.11.0
+sqlglotc==30.11.0          typing-extensions==4.16.0
+typing-inspection==0.4.5
+```
+
+Binary-only resolution of this complete closure also succeeded for Python 3.14
+on Linux x86-64 (`x86_64-manylinux_2_28`). This checks wheel availability, not
+execution on that architecture. An attempted Linux x86-64 container run failed
+in QEMU while starting the tooling, before Quarry checks could execute.
+
+Native Linux ARM64 diagnostics also passed on CPython 3.14.10: all 54 Quarry
+tests, lint, typecheck, offline example, and wheel/sdist build. That disposable
+container explicitly used pip's `--ignore-requires-python` and uv's
+`UV_NO_SYNC=1`. Its installed upstream metadata still excluded 3.14. The
+container's uv dependency checker did not flag that Python restriction, so the
+audit also checked it explicitly with `packaging.specifiers.SpecifierSet`;
+the result was incompatible. Installer success is not support certification.
+
+## Checks performed
+
+Existing Makefile targets were used with isolated `/tmp` environments and public
+PyPI. The normal workspace installation contains no Quarry on 3.14. PostgreSQL
+16 was available for the server and SDK integration suites.
+
+| CPython 3.14.7 on macOS ARM64 | Result |
+| --- | --- |
+| Workspace `make sync` and `uv pip check` | Passed; all installed packages compatible. |
+| Models | 76 passed. |
+| Telemetry | 20 passed. |
+| Engine | 138 passed. |
+| Builtin evaluators | 311 passed. |
+| Server | 1,040 passed. |
+| Python SDK | 754 passed, 2 skipped because optional Galileo extras were absent. |
+| Root packaging/build scripts | 6 passed. |
+| Budget / Cisco / DefenseClaw / Galileo | 122 / 23 / 26 / 358 passed; package lint and typecheck passed. |
+| Core lint | Passed. |
+| Core typecheck with mypy 2.4.1 | All 116 source files passed. |
+| Quarry, diagnostic installation only | 54 passed; lint, typecheck, offline example, and wheel/sdist build passed. |
+| Normal Quarry installation | Blocked by `>=3.12,<3.14`. |
+| macOS diagnostic environment dependency check | Fails as expected for local Quarry and `cisco-project-quarry`; runtime success does not remove this incompatibility. |
+
+One server test had assumed that valid JSON nested 10,000 levels always raises
+a decoder recursion error. Python 3.14 can decode it. The test now accepts either
+an accurate parsed count or an undecodable summary, while requiring the same
+502 rejection and empty sanitized error details. All 1,040 server tests also
+passed on Python 3.12.13 after this test-only adjustment. Server runtime behavior
+was not changed.
+
+## Lockfiles and reproducibility
+
+Python `uv.lock` files are ignored by this repository, including the root and
+contrib lockfiles. No Python lock is published by this change. The local root
+lock admits `>=3.12,<3.15`; Quarry's local lock retains `>=3.12,<3.14`.
+
+The existing local root lock selected mypy 2.1.0, which reports two `TypeAdapter`
+alias errors with Pydantic 2.14.1. The same errors reproduce on 3.12, so they are
+not a Python 3.14 runtime regression. Mypy 2.4.1 passes the entire core tree.
+A clean dependency resolution was tested separately from the existing local
+lock; this distinction matters when reproducing CI from an ignored lockfile.
+That clean resolution selected 89 installed packages, including mypy 2.4.1 and
+Pydantic 2.14.1, and passed the complete core check command:
+
+```sh
+UV_INDEX=https://pypi.org/simple UV_PYTHON=3.14 \
+  UV_PROJECT_ENVIRONMENT=/tmp/agent-control-clean314-env \
+  make check -o contrib-test -o contrib-lint -o contrib-typecheck
+```
+
+The non-Quarry contrib packages were checked independently in separate 3.14
+environments. The exclusions above isolate their package environments and the
+known Quarry blocker; they do not replace those independent contrib checks.
+The CI YAML was parsed and its interpreter selection verified locally; the
+hosted GitHub workflow itself has not been run as part of this audit.
+
+The documented diagnostic Quarry checks used `UV_NO_SYNC=1` after explicitly
+installing the incompatible distributions into a disposable environment. This
+is a compatibility experiment, not an installation recommendation.
+
+## Requirements for future Python 3.14 Quarry support
+
+These requirements concern a future 3.14 Quarry deployment. They do not block
+the supported Python 3.13 deployment described above.
+
+1. Obtain a reviewed upstream release or immutable fork that declares Python
+   3.14 support and passes the compiler/scanner tests and dependency checks.
+2. Update our upstream dependency pin, widen the evaluator's Python range, and
+   update the builtin extra's interpreter marker together.
+3. Re-run normal installs, all Quarry checks, and the complete workspace checks
+   without metadata bypasses or the explicit CI Quarry exclusion. Validate the
+   actual Orbit Linux/Bazel environment; x86-64 wheel resolution is insufficient.
+4. Commit and normally push that validated change, then provide its full SHA.
+   The current implementation baseline does not satisfy these 3.14 conditions.