Skip to content

feat(guardrails): group guardrail checks into stage containers - #1222

Open
daniel-joica wants to merge 3 commits into
mainfrom
feat/guardrail-stage-grouping
Open

daniel-joica wants to merge 3 commits into
mainfrom
feat/guardrail-stage-grouping

Conversation

@daniel-joica

@daniel-joica daniel-joica commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

3 of 3 (#1220 and #1221 are merged; this is now based on main):

  1. feat(guardrails): record each guardrail action as a span [AL-712] #1220 (merged): a span for each guardrail action.
  2. feat(guardrails): record deterministic rule checks as spans #1221 (merged): a span for each deterministic rule check.
  3. This PR: grouping checks into stage containers.

Works with the published uipath-platform; no platform change needed. Together with UiPath/uipath-python#1997, which removes the duplicate client evaluate_guardrail span.

What

Each guardrail middleware hook is its own LangGraph node, so seven guardrails gave about seven separate guardrail nodes around each model call. Low-code agents group a stage's checks into one container. This PR does the same for coded agents, without changing how guardrails are written (middleware=[*UiPathPIIDetectionMiddleware(...), ...]):

LangGraph
├─ LLM input guardrail check   (llmPreGuardrails)   ← shared by 4 guardrail nodes
│  ├─ PII Detection             (Guardrails Service)
│  ├─ Log                       (guardrailAction, #1220)
│  ├─ Harmful Content Detection
│  └─ ...
├─ model                        ← its start closes the container
├─ tools
│  ├─ Tool input guardrail check  → Tool PII Detection, London Weather Rule (#1221), Log
│  ├─ get_weather
│  └─ Tool output guardrail check
└─ LLM output guardrail check / Agent output guardrail check

Container names and span types are the low-code ones (agentPre/Post, llmPre/Post, toolPre/PostGuardrails).

How

In guardrails/middlewares/_spans.py:

  • guardrail_stage(scope, stage) wraps each check (evaluation and action). The first check of a stage creates a shared container next to the guardrail nodes (inside tools for tool checks); later guardrail nodes of that stage reuse it. It's lazy: a stage where every check is skipped leaves nothing.
  • guardrail_aware(provider) wraps the runtime's current span provider so a running check reports its container as the current span. @traced spans, the action span and the Guardrails Service's span (through the trace headers) all land in it.
  • GuardrailStageCloser (span processor) ends a container when a sibling that isn't this stage's guardrail node starts (at that sibling's start time, but never before its last check ended: coarse clocks such as Windows before Python 3.13 can stamp the sibling a tick early), or when its parent ends.
  • is_exported (span filter in the factory's trace settings) hides the now-empty guardrail node spans: only OpenInference spans with a registered guardrail node name, only while grouping is on.
  • Containers copy execution.id from the node (their parent is non-recording), so execution-based lookups (evals) include them.
  • A container that closes without an error gets an explicit OK status, like the action and evaluation spans (feat(guardrails): record each guardrail action as a span [AL-712] #1220 review); a failed one keeps ERROR.
  • Creating a stage's container also closes the other stage's container under the same parent, for when no sibling starts in between (a middleware that answers a tool call without running the tool).
  • If a container can't be created, the check runs ungrouped and a warning is logged, so grouping never skips a guardrail.
  • No grouping under a non-recording (sampled-out) span, whose end never reaches the closer.

Opt-in by use: runtime/factory.py calls enable_guardrail_grouping(trace_manager, get_current_span), which only registers the provider wrapper and offers the trace manager. A closer is attached, and grouping turns on, once a guardrail middleware is built (guardrails_in_use()). Agents without guardrails, and low-code agents (whose factory subclasses this one), get no closer and no filtering. When the trace manager shuts down, the closer unregisters itself, forgets that manager and ends anything still open; the next run adds a new one.

Other changes:

  • _base.py: checks run in guardrail_stage; message hooks register their node names; the tool wrapper signals use.
  • deterministic.py: the rule check runs in the tool stage, and building the middleware signals use.
  • test_action_span.py: each test starts with fresh grouping state.
  • Version 0.20.7 → 0.20.8.

Known limits

  • Anything that runs between two guardrail nodes (another middleware) starts a new container for the rest of that stage.
  • LLM checks sit next to model, not inside an LLM call span as in low-code: before_model, model and after_model are separate nodes. Wrapping the model call (wrap_model_call) could add that level later.
  • A custom action that runs a LangChain runnable inside a guardrail hook creates spans under the hidden node, and in a tool stage it ends the container before the check finishes.
  • is_exported hides guardrail node spans by name across the process, so another graph's runnable with exactly that name would be hidden too.
  • If grouping fails for a check, its spans fall back to the guardrail node, which is already hidden (the decision is made when the node starts).
  • If two trace managers are alive at once, one's shutdown() clears the shared processor list (uipath-core), which also silences the other's closer. Today's CLI flows use one at a time.

Tests

  • New test_grouping.py (27):
    • stages: nodes of one stage share a container that ends exactly at model's start, and never before its last check; execution.id copied; a new stage gets its own container; a node in between splits it; LLM output; a block marks the container failed; a grouping failure still runs the check; nothing to check means no container;
    • tools: input and output containers with exact boundaries; the input container closes when the output stage starts without a tool span, and stays closable if the output stage can't start;
    • enabling: off without a closer, under an invalid span, and without guardrail middlewares; one closer per trace manager, back after shutdown, a shut-down manager isn't reused; nothing grouped under a sampled-out node; no deadlock when attribute limits drop the container's mark; tool-only guardrails turn it on; only OpenInference guardrail node spans are hidden;
    • real create_agent + OpenInference: 2 parallel tool calls, 3 parallel tool calls with deliberately overlapping checks, and 2 concurrent runs (no orphans, nothing left open, every check inside its container).
  • Full suite: 4376 passed, 3 skipped. ruff, ruff format and mypy are clean.
  • End to end: jobs on a local robot (an agent with 7 guardrails) show the structure above in LLMOps, with every service evaluation inside its container and no guardrail node spans exported.
  • Reviewed by three independent reviewers before this split; their findings are fixed or listed under Known limits. feat(guardrails): group coded-agent guardrail spans like low-code agents [AL-712] #1219 holds the earlier, single-PR version and its review thread.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings November 18, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A tracing failure can bypass blocking guardrails, and unresolved span lifecycle and filtering defects can corrupt traces.

5 open issues
What changed in this PR

Completes the guardrail tracing work by grouping coded-agent checks into stage containers aligned with low-code traces, without changing middleware usage.

Changes:

  • Adds shared stage spans, automatic closure, and guardrail-node filtering.
  • Integrates grouping with runtime setup and built-in and deterministic checks.
  • Adds grouping tests, isolates action-span test state, and bumps the version to 0.20.5.
File Description
uv.lock Updates the locked package version to 0.20.5.
tests/​guardrails/​middlewares/​test_grouping.py Tests grouping, boundaries, activation, filtering, and concurrent runs.
tests/​guardrails/​middlewares/​test_action_span.py Resets grouping state for test isolation.
src/​uipath_langchain/​runtime/​factory.py Registers grouping support and the span filter.
src/​uipath_langchain/​guardrails/​middlewares/​deterministic.py Wraps deterministic tool checks in stage containers.
src/​uipath_langchain/​guardrails/​middlewares/​_spans.py Implements container sharing, parenting, closure, and filtering.
src/​uipath_langchain/​guardrails/​middlewares/​_base.py Registers guardrail hooks and wraps built-in checks in stages.
pyproject.toml Bumps the package version to 0.20.5.

Review effort: Balanced

@daniel-joica
daniel-joica force-pushed the feat/guardrail-deterministic-spans branch from 13cf3c6 to 6fa85ec Compare November 19, 2026 13:00
Base automatically changed from feat/guardrail-deterministic-spans to main November 20, 2026 08:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daniel-joica
daniel-joica force-pushed the feat/guardrail-stage-grouping branch from 4e9d992 to a003aa0 Compare November 20, 2026 08:47
@daniel-joica daniel-joica changed the title feat(guardrails): group guardrail checks into stage containers [AL-712] feat(guardrails): group guardrail checks into stage containers Nov 20, 2026
…g goes wrong

- A failure to create a stage container now runs the check ungrouped.
- Creating a stage's container closes the other stage's container under
  the same parent.
- Shutting down a trace manager also forgets it, so it isn't reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…ntainers in grouping

- The closer skips a starting container through a context flag, not only
  its attribute, so it never re-enters the lock.
- Grouping is skipped under a non-recording (sampled-out) span.
- A new stage's container is created before the previous one is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shared parent IDs can mix traces, cancellation can close stages before checks finish, and span tracking can leak containers or serialize callbacks across runs.

No open issues 🎉

4 previously missed issues

In code that hasn't changed since last review

Medium severity Keep guardrail stage open until cancelled worker finishes

src/​uipath_langchain/​guardrails/​middlewares/​_base.py:427

[P2] Cancelling this asyncio.to_thread await exits guardrail_stage, but the worker keeps running with its copied container context. The tools span can then close the container as OK before the service evaluation finishes, leaving a child outside its stage's lifetime. Keep the stage open until the worker actually finishes on cancellation; apply the same handling to the POST check at _base.py:469.

Medium severity Clean up containers whose unsampled parent never ends

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:235

[P2] A valid parent context does not imply the parent span is recording. With a name-based sampler that records this guardrail node and the container but drops its graph parent and subsequent siblings, the container is cached under a parent that will never trigger on_end, and no sampled sibling triggers on_start; the container and its last-check entry then remain until shutdown. Verify that the parent can report its end, or provide a node-end fallback when it cannot.

Medium severity Scope container keys and bookkeeping by trace ID

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:238

[P2] Two concurrent traces can legally have the same parent span ID, for example with a custom ID generator. This key then reuses one trace's container for the other, and either parent can close it. Use (trace_id, span_id, span_type) keys and trace-scoped last-check bookkeeping in every creation and close path.

Medium severity Index containers by parent to avoid global scans

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:327

[P2] Most span-start callbacks and every span-end callback scan all active containers under the process-wide lock, including spans from unrelated runs. With many overlapping checks, total tracing work can become quadratic and callbacks serialize across runs. Index containers by parent context so each callback only examines that parent's stages.

4 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants