Skip to content

feat(guardrails): group coded-agent guardrail spans like low-code agents [AL-712] - #1219

Open
daniel-joica wants to merge 3 commits into
mainfrom
feat/guardrail-grouping-in-tracing
Open

daniel-joica wants to merge 3 commits into
mainfrom
feat/guardrail-grouping-in-tracing

Conversation

@daniel-joica

@daniel-joica daniel-joica commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Part of AL-712, together with UiPath/uipath-python#1997, which removes the duplicate client evaluate_guardrail span. This PR works with the published uipath-platform; it needs no platform change.

What

In a coded agent, every guardrail middleware hook is its own LangGraph node. Seven guardrails gave about seven guardrail nodes around each model call: no grouping, no record of the action taken, and nothing at all for deterministic rules. Low-code agents group checks into stage containers and record each action.

This PR makes coded traces match low-code's, without changing how guardrails are written (middleware=[*UiPathPIIDetectionMiddleware(...), ...]):

LangGraph
├─ LLM input guardrail check   (llmPreGuardrails)   ← shared by 4 guardrail nodes
│  ├─ PII Detection             (Guardrails Service)
│  ├─ Log                       (guardrailAction, linked to PII Detection)
│  ├─ Harmful Content Detection
│  └─ ...
├─ model                        ← its start closes the container
├─ tools
│  ├─ Tool input guardrail check  → Tool PII Detection, London Weather Rule (custom), Log
│  ├─ get_weather
│  └─ Tool output guardrail check
└─ LLM output guardrail check / Agent output guardrail check

Container names and span types are the low-code ones (agentPre/Post, llmPre/Post, toolPre/PostGuardrails).

How

All logic is in the new guardrails/middlewares/_spans.py:

  • guardrail_stage(scope, stage) wraps each check (evaluation and action). The first check of a stage creates a shared container next to the guardrail nodes (inside tools for tool checks); later guardrail nodes of that stage reuse it. It's lazy: a stage where every check is skipped leaves nothing.
  • guardrail_aware(provider) wraps the runtime's current span provider so a running check reports its container as the current span. @traced spans, the action span and the Guardrails Service's span (through the trace headers) all land in it.
  • GuardrailStageCloser (span processor) ends a container when a sibling that isn't this stage's guardrail node starts (at that sibling's start time, but never before its last check ended: coarse clocks such as Windows before Python 3.13 can stamp the sibling a tick early), or when its parent ends.
  • is_exported (span filter in the factory's trace settings) hides the now-empty guardrail node spans: only OpenInference spans with a registered guardrail node name, only while grouping is on.
  • Action span (guardrailAction: Block, Log, ...) with guardrailName, reason, severityLevel, and guardrailSpanId linking it to its evaluation.
  • Deterministic rules get a client guardrailEvaluation span (guardrailType=custom, validationResult, details), and their action links to it.
  • Containers copy execution.id from the node (their parent is non-recording), so execution-based lookups (evals) include them.

Opt-in by use: runtime/factory.py calls enable_guardrail_grouping(trace_manager, get_current_span), which only registers the provider wrapper and offers the trace manager. A closer is attached, and grouping turns on, once a guardrail middleware is built (guardrails_in_use()). Agents without guardrails, and low-code agents (whose factory subclasses this one), get no closer and no filtering. When the trace manager shuts down, the closer unregisters itself and ends anything still open; the next run adds a new one.

Other changes:

  • _base.py: checks run in guardrail_stage, actions in guardrail_action_span; message hooks register their node names.
  • deterministic.py: a _check method (evaluation span, then action) inside the tool stage. Behaviour is unchanged.
  • Version 0.20.2 → 0.20.3.

Known limits

  • Anything that runs between two guardrail nodes (another middleware) starts a new container for the rest of that stage.
  • LLM checks sit next to model, not inside an LLM call span as in low-code: before_model, model and after_model are separate nodes. Wrapping the model call (wrap_model_call) could add that level later.
  • A custom action that runs a LangChain runnable inside a guardrail hook creates spans under the hidden node.
  • If two trace managers are alive at once, one's shutdown() clears the shared processor list (uipath-core), which also silences the other's closer. Today's CLI flows use one at a time.

Tests

  • test_action_span.py (8): Block error span with all attributes, linked and parented; Log with severity; no span on a pass; an escalation pause is not an error; missing service id omitted; action names.
  • test_grouping.py (21): nodes of one stage share a container that ends exactly at model's start, and never before its last check; execution.id copied; a new stage gets its own container; a node in between splits it; LLM output; a block marks the container failed; nothing to check means no container; tool input and output containers with exact boundaries; off without a closer, under an invalid span, and without guardrail middlewares; one closer per trace manager, back after shutdown; tool-only guardrails turn it on; only OpenInference guardrail node spans are hidden; real create_agent + OpenInference with 2 parallel tool calls and 2 concurrent runs (no orphans, nothing left open), and 3 parallel tool calls with deliberately overlapping checks (each call has its own tools span, so its own container, and every check ends inside it).
  • Full suite: 4322 passed, 3 skipped. ruff, ruff format, mypy and the httpx lint are clean.
  • End to end: jobs on a local robot (an agent with 7 guardrails) show the structure above in LLMOps, with every service evaluation inside its container and no guardrail node spans exported.
  • Three independent reviews; their findings are fixed or listed under Known limits.

🤖 Generated with Claude Code

Each guardrail middleware hook is its own LangGraph node, so a coded agent's
trace showed one node per guardrail per hook, with no grouping, no action spans
and nothing for deterministic rules.

- Stage containers ("LLM input guardrail check", ...) shared across the separate
  guardrail nodes: the first check of a stage opens one, later guardrail nodes
  of that stage reuse it, the next unrelated node closes it. The empty guardrail
  node spans are hidden from the export.
- A guardrailAction span (Block, Log, ...) per action, linked to its evaluation.
- A client guardrailEvaluation span for deterministic rules.

The API is unchanged. Grouping only turns on when a guardrail middleware is in
use and a trace manager is available.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 09:57
@daniel-joica daniel-joica changed the title feat(guardrails): group coded-agent guardrail spans like low-code agents feat(guardrails): group coded-agent guardrail spans like low-code agents [AL-712] Nov 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Overlapping guardrail evaluations can cause a shared tool-stage container to close prematurely, leaving slower checks outside their parent span and preventing late Block results from marking it as failed.

1 open issue
What changed in this PR

This PR aligns coded-agent guardrail traces with low-code agents by grouping checks into stage containers, without changing how middleware is configured.

Changes:

  • Adds shared stage containers, lifecycle handling, and filtering of redundant guardrail node spans.
  • Records guardrail actions and deterministic evaluations, with tracing tests.
  • Integrates grouping into the runtime factory and bumps the version to 0.20.3.
File Description
uv.lock Updates the package version to 0.20.3.
tests/​guardrails/​middlewares/​test_grouping.py Tests grouping, lifecycle, filtering, and concurrent runs.
tests/​guardrails/​middlewares/​test_action_span.py Tests action attributes, linkage, errors, and interrupts.
src/​uipath_langchain/​runtime/​factory.py Registers grouping and the span filter.
src/​uipath_langchain/​guardrails/​middlewares/​deterministic.py Adds evaluation and action tracing to deterministic checks.
src/​uipath_langchain/​guardrails/​middlewares/​_spans.py Implements guardrail spans and shared container lifecycle.
src/​uipath_langchain/​guardrails/​middlewares/​_base.py Wraps checks and actions and registers guardrail nodes.
pyproject.toml Bumps the project version to 0.20.3.

Review effort: Balanced

return None # nothing would ever close it

name, span_type = _STAGES[(scope, stage)] # type: ignore[index]
key = (parent.span_id, span_type)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In create_agent, LangGraph runs each tool call in its own tools span (one per Send), and tool containers are keyed by that span, so parallel calls never share one. Added test_overlapping_checks_on_parallel_tool_calls_stay_in_their_container (3 parallel calls, deliberately overlapping 50 ms checks): three containers, every check and action ends inside its own.

daniel-joica and others added 2 commits November 18, 2026 12:51
…-712]

Addresses review on #1219: a regression test with three parallel tool calls and
deliberately overlapping checks (each call has its own tools span, so its own
container). Also Sonar: collect the closer's owners before deleting them, one
throwing call per pytest.raises, and split a composite assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…L-712]

On Windows before Python 3.13 the clock advances about every 15 ms, so the next
sibling's start can be stamped a tick before the last check ended. Each stage
now records when its last check finished, and the closer ends the container at
the later of the two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants