Skip to content

feat(llamaindex): let the LLM Gateway record model spans - #383

Open
daniel-joica wants to merge 5 commits into
mainfrom
feat/llamaindex-gateway-trace-context
Open

daniel-joica wants to merge 5 commits into
mainfrom
feat/llamaindex-gateway-trace-context

Conversation

@daniel-joica

@daniel-joica daniel-joica commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Brings the LLM Gateway trace linking from uipath-langchain (uipath-langchain-python#1197) to uipath-llamaindex.

  • Trace-context headers: every UiPath model (UiPathOpenAI, UiPathChatBedrock / UiPathChatBedrockConverse, UiPathVertex) now sends x-uipath-traceparent-id and x-uipath-tracebaggage on each Gateway request. The Gateway then records the model call ("Model run", with its governance and guardrail spans) inside the agent's trace. Before this, LlamaIndex models sent no trace context.
  • No duplicate client span: OpenInference's client-side LLM span would duplicate "Model run", so it is no longer created. It is never started, not just dropped at export, so live tracking can't leave it behind as "running".
  • Opt-out: UIPATH_USE_CLIENT_LLM_SPANS=true (also 1 / yes) keeps the client-side LLM span and stops sending the traceparent; the baggage is still sent. This is the same variable as uipath-langchain, for agents whose models don't go through the Gateway. It is read on every call.

How

  • _trace_context.py: the env var and trace_context_headers(), called from each model's existing per-request hook (OpenAI's httpx transport, Bedrock's _modify_request, Vertex's _rewrite_request_for_gateway). Headers are built fresh for each request from the active span. If building them fails, the request goes out without them.

  • runtime/_llm_spans.py: LlamaIndex nests spans by its own span ids, not OTel context, so an LLM call without a span needs three things:

    • whatever it calls nests under its nearest ancestor that has a span;
    • its exit and events are ignored, rather than logged as Open span is missing;
    • current_span() (the provider registered with UiPathSpanUtils) gives the traceparent that ancestor's span, so the Gateway's spans land under the workflow step or agent that made the call.

    These are subclasses of OpenInference's span and event handlers. skip_client_llm_spans() switches the instances instrument() built to them. A skipped call is kept only while it runs: it is forgotten when it exits or is dropped, which llama-index always reports, streams included. A chat or completion stream still emits events with the call's id after it returns (in-progress, LLM…EndEvent, ExceptionEvent). Those are ignored when OpenInference has no span for that id, so nothing is kept for a stream, and a stream that is never read can't leak.

  • runtime/factory.py: installs both.

Dependencies

uipath-platform>=0.3.2, <0.4.0 is the first release where the trace-context headers aren't behind the EnableTraceContextHeaders flag (uipath-python#1979). That requires uipath>=2.15.1, <2.16.0. The lock uses uipath 2.15.1 and uipath-platform 0.3.2 rather than today's 2.15.2 / 0.3.4, which CI's package proxy rejects as too new. The version is bumped to 0.6.3.

Tests

tests/runtime/test_trace_context.py (30 tests) runs a real workflow against a local tracer provider. It covers:

  • no client LLM span by default;
  • the traceparent names the calling step's span;
  • no missing-span warnings;
  • errors still propagate;
  • the env var values (true / TRUE / 1 / yes keep the span, false / 0 / empty skip it);
  • cleanup after normal, failing and streamed calls, including streams abandoned half-way or never read (async and sync);
  • late stream events of an unknown span are ignored, while other events of an unknown span are still reported as missing;
  • idempotence and factory wiring;
  • the headers on each of the three models.

Each mechanism was checked by breaking it on purpose, and the tests fail when it is broken.

testcases/template-agent/expected_traces.json no longer requires the client LLM span, which is gone by default. The same change was made for uipath-langchain.

Notes for reviewers

  • The span handling depends on OpenInference's private _SpanHandler / EventHandler. test_factory_installs_the_skip fails if an upgrade renames them, so client LLM spans can't silently come back.
  • Agents using a non-UiPath LlamaIndex model (e.g. native OpenAI) have no Gateway span to replace the client one. They lose their LLM span by default unless they set UIPATH_USE_CLIENT_LLM_SPANS=true. This matches uipath-langchain.
  • Embeddings are unchanged: they send no traceparent and keep their client spans.
  • The baggage source is source=llamaindex, following source=claude-sdk.

QA

Local robot QA will follow in a comment.

uipath-llamaindex/chat-models fails for UiPathChatBedrock only (Cannot instantiate typing.Union). This also happens on main and comes from anthropic 2.x (released Oct 21): llama-index-llms-bedrock pulls in llama-index-llms-anthropic 0.8.x, which calls anthropic.types.MessageParam(...), and MessageParam is now a Union. It is unrelated to this change.

🤖 Generated with Claude Code

UiPath LlamaIndex models (OpenAI, Bedrock, Vertex) now send the LLM Gateway
trace-context headers on every request, so the Gateway records the model call
("Model run", with its governance and guardrail spans) inside the agent's
trace. OpenInference's client-side LLM span would duplicate it, so the runtime
no longer creates one.

UIPATH_USE_CLIENT_LLM_SPANS=true keeps the client-side LLM span instead and
stops sending the traceparent (the baggage is still sent), for agents whose
models don't go through the Gateway. Same variable as uipath-langchain.

Requires uipath-platform>=0.3.2, where the trace-context headers are no longer
behind the EnableTraceContextHeaders flag, and so uipath>=2.15.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 12:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The async-stream finalizer removes skipped-call registration before closure finishes, allowing cleanup events to reach OpenInference and produce missing-span warnings when a partially consumed stream is garbage collected.

1 open issue
What changed in this PR

This PR links LlamaIndex model requests to UiPath LLM Gateway traces, making Gateway-recorded model spans the default instead of duplicate client-side spans.

Changes:

  • Adds per-request trace headers to OpenAI, Bedrock, and Vertex models.
  • Suppresses client LLM spans while preserving ancestor context, with an environment-variable opt-out.
  • Adds tracing tests, updates expected traces, and bumps the package and required dependencies.
File Description
packages/​uipath-llamaindex/​uv.lock Locks updated package and dependency versions.
packages/​uipath-llamaindex/​tests/​runtime/​test_trace_context.py Tests headers, span suppression, opt-out, and cleanup.
packages/​uipath-llamaindex/​testcases/​template-agent/​expected_traces.json Removes the expected client LLM span.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​runtime/​factory.py Installs span suppression and ancestor-context lookup.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​runtime/​_llm_spans.py Handles skipped LLM spans, events, and parent context.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​llms/​vertex.py Adds trace headers to Vertex Gateway requests.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​llms/​bedrock.py Adds trace headers to Bedrock Gateway requests.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​llms/​_openai.py Adds trace headers to synchronous and asynchronous OpenAI requests.
packages/​uipath-llamaindex/​src/​uipath_llamaindex/​_trace_context.py Centralizes trace-header construction and opt-out handling.
packages/​uipath-llamaindex/​pyproject.toml Bumps to 0.6.3 and raises UiPath dependency requirements.

Review effort: Balanced

daniel-joica and others added 2 commits November 10, 2026 15:08
CI's package proxy rejects releases younger than its minimum age, and
uipath 2.15.2 and uipath-platform 0.3.4 were published today. 0.3.2 is the
first platform release without the EnableTraceContextHeaders flag, and
uipath 2.15.1 accepts it, so the floor moves to uipath>=2.15.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A half-read stream that was dropped lost its registration before it was
closed: CPython clears weak references before closing a generator, and
llama-index's stream wrapper emits an ExceptionEvent (GeneratorExit) while
closing, which then reached OpenInference as a missing span.

A skipped call is now forgotten when it returns or fails, or, for a chat or
completion stream, when its final event (end or exception) arrives, which the
wrapper emits exactly once when the stream is exhausted, fails or is closed.

Also builds the failing model outside pytest.raises in the tests (Sonar S5778).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
daniel-joica and others added 2 commits November 10, 2026 16:30
A stream that was returned but never read sends no further events, so its
registration, kept until the stream's final event, was never removed: one
entry leaked per abandoned stream.

A skipped call is now forgotten when it exits or is dropped, which
llama-index always reports, streams included. The events a stream emits
afterwards (in-progress, end, exception) are ignored when OpenInference has
no span for their id, so nothing is kept for a stream at all.

Tests cover both sides of that filter: late stream events of an unknown span
log nothing, and other events of an unknown span are still reported as
missing. They read OpenInference's logger directly rather than through
caplog, which other tests in the suite leave disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Custom HTTP clients can bypass trace-header injection while the client LLM span is still suppressed, leaving model calls unlinked from the agent trace.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Custom HTTP clients can break trace propagation while suppressing LLM spans

packages/​uipath-llamaindex/​src/​uipath_llamaindex/​runtime/​_llm_spans.py:105

UiPathOpenAI lets http_client and async_http_client supplied through **kwargs replace the clients containing the trace-header transports. With a custom client that does not inject these headers, this check still suppresses the client LLM span, so the model call is no longer linked to the agent trace. Inject fresh trace headers for supplied clients too, or retain the client span when propagation is unavailable, and cover both overrides with regression tests.

1 resolved since last review

Review effort: Balanced

@daniel-joica

Copy link
Copy Markdown
Contributor Author

Re Copilot's "Custom HTTP clients can break trace propagation while suppressing LLM spans" (_llm_spans.py:105):

That's right: UiPathOpenAI applies {**defaults, **kwargs}, so a caller's http_client / async_http_client replaces the UiPath transports and the traceparent isn't sent. But those transports also rewrite /openai/deployments/... to the Gateway's /completions path (llms/_openai.py:22-29), and that was already lost with a custom client before this PR. So custom clients aren't supported today, and the missing trace link is a side effect of that, not something this PR introduces.

Leaving it out of scope here. If custom clients should be supported, a follow-up can add both the headers and the URL rewrite to whatever client is passed (e.g. through httpx event_hooks), with tests for both overrides.

@daniel-joica

daniel-joica commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

QA:

  • UIPATH_USE_CLIENT_LLM_SPANS=false (default state)
Screenshot 2026-11-10 at 17 21 13
  • UIPATH_USE_CLIENT_LLM_SPANS=true
Screenshot 2026-11-10 at 17 21 59
  • current alpha cloud:
Screenshot 2026-11-10 at 17 22 41_cloud_alpha

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants