feat(tracing): add W3C distributed tracing and incremental handler batching

[pradystar]

Summary

This PR completes the SDK’s W3C distributed-tracing migration and satisfies both distributed-tracing requirements:

• DT.A — Distributed context propagation: SDK-native telemetry, native OpenTelemetry spans, and supported upstream OTel HTTP instrumentation can participate in the same trace across service boundaries.
• DT.B — Incremental span delivery: Completed native-handler operations enter the existing OpenTelemetry BatchSpanProcessor when their callbacks end without waiting for the entire logical trace to complete.

Both automatic and explicit propagation remain supported.

Detailed requirements mapping, lifecycle behavior, compatibility analysis, deletion rationale, and validation evidence are available in the detailed review context PR_REVIEW_DETAILS.md

What changed

Standard W3C propagation

Distributed context now uses:

• traceparent
• tracestate
• W3C baggage

This replaces the proprietary Splunk-AO-Trace-ID and Splunk-AO-Parent-ID propagation format and allows Splunk AO telemetry to interoperate with upstream OpenTelemetry instrumentation and non-Python services.

Automatic distributed tracing

Applications can configure supported automatic instrumentation with:

from fastapi import FastAPI
from splunk_ao import configure_distributed_tracing

app = FastAPI()
tracer_provider = configure_distributed_tracing(app=app)

This:

• Creates or accepts a concrete TracerProvider.
• Registers the Splunk AO span processor once.
• Enables installed supported HTTP instrumentors.
• Instruments the supplied FastAPI or Starlette application.
• Returns the application-owned provider for normal shutdown.
• Does not replace the process-global tracer provider.

Install the optional dependencies with:

pip install "splunk-ao[distributed-tracing]"

Automatic incoming extraction is supported for FastAPI and Starlette.

Automatic outgoing injection is supported for:

• Requests
• HTTPX sync and async clients
• aiohttp

Applications using other frameworks or custom transports can register the corresponding upstream OTel instrumentor or continue using explicit propagation.

Explicit propagation remains supported

Existing explicit integrations can continue to use:

from splunk_ao import get_tracing_headers

Incoming context can continue to use extract_tracing_context() or TracingMiddleware.

These APIs now use the same W3C context as automatic instrumentation. The explicit approach is not deprecated.

Incremental native-handler completion

LangChain, CrewAI, Google ADK, and OpenAI Agents operations now enter the existing span-processing pipeline when their individual callbacks end.

A completed child can therefore become eligible for scheduled or size-based BSP export while its parent agent or workflow remains active.

This does not mean every callback causes an immediate network request. The existing BatchSpanProcessor still determines network-export timing according to standard BSP configuration.

No per-trace flush() is required, and flush() does not end active work.

Control-plane HTTP suppression

SDK-owned authentication, health-check, routing, CRUD, token-refresh, and related HTTP operations are scoped under standard OTel HTTP suppression.

This prevents SDK control-plane calls from appearing as unrelated application GET or POST traces when automatic HTTP instrumentation is enabled.

Suppression is scoped to the SDK request and does not suppress application HTTP traffic.

Compatibility guarantees

This change preserves:

• Existing single-service parent-child relationships.
• Existing local trace structure with and without distributed tracing.
• Logger, @log, and native handler instrumentation.
• SDK-native and caller-owned OTel instrumentation.
• Existing standalone and O11y deployment routing.
• Caller ownership of caller-created providers.
• The rule that the internal trace envelope is not exported as a span.
• The behavior that flush() drains completed work without ending active operations.
• ingestion_hook whole-tree compatibility.
• flush_on_chain_end compatibility.

Distributed tracing adds remote ancestry without otherwise reshaping the application’s local telemetry tree.

start_new_trace remains a handler ownership option, not a distributed-tracing switch. Users do not need to change its default value to enable distributed tracing.

Session propagation and privacy

An explicit session is propagated through W3C baggage as:

gen_ai.conversation.id

The implementation does not propagate project, Agent Stream, agent, experiment, application, routing, endpoint, authentication, prompt, response, or embedding data through baggage.

Intentional breaking changes

The migration removes the proprietary propagation surface:

• trace_id= and span_id= arguments on SplunkAOLogger. Note that all example might not be updated and examples might be stale. The examples will be fixed in a future PR.
• The logger instance method get_tracing_headers().
• Splunk-AO-Trace-ID and Splunk-AO-Parent-ID.

Applications should use the module-level W3C helper:

from splunk_ao import get_tracing_headers

or enable automatic propagation through configure_distributed_tracing().

Removed delivery path

The previous distributed-mode REST task handler maintained a second queue, dependency-ordering system, retry lifecycle, and shutdown path.

Normally exported telemetry now follows one processing path:

completed operation
→ SpanSink
→ OpenTelemetry BatchSpanProcessor
→ deployment-aware OTLP exporter

The obsolete task handler and its dedicated tests were removed. This does not remove any logger, decorator, or native framework-handler support.

Validation summary

Completed validation includes:

• Full root suite: 2,163 passed, 4 skipped
• LangChain and CrewAI compatibility: 202 passed
• Google ADK suite: 252 passed
• A2A suite: 68 passed
• Final focused root regression set: 212 passed
• Automatic instrumentation tests: 20 passed
• Control-plane suppression and O11y HTTP tests: 41 passed
• Ruff lint and formatting checks: passed
• Mypy: 118 source files passed
• Poetry lock consistency: passed
• git diff --check: passed

Live standalone validation succeeded for both automatic and explicit cross-service distributed tracing.

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-sonnet-5). It may contain mistakes.

Verdict: request_changes — A concrete id-reuse aliasing bug in the new automatic-instrumentation bookkeeping (http_instrumentation.py) can cause silent double-instrumentation, false conflict errors, or unbounded growth of module-level state; there are also stale/incorrect docs left over from the removed proprietary header scheme.

General Comments

• 🟠 major (bug): http_instrumentation.py tracks instrumentation state using id(app) / id(tracer_provider) as keys in plain module-level dict/set structures (_client_provider_ids, _instrumented_apps) without holding any reference (strong or weak) to the underlying objects. _configured_providers correctly uses a WeakSet to avoid this problem, but the other two do not. Once an app/tracer_provider is garbage-collected, CPython can and does reuse its id() for an unrelated object; a subsequent unrelated app/provider can then collide with a stale entry, causing _instrument_app/_validate_client_ownership to silently skip instrumentation (treating the new object as 'already instrumented') or to raise a false 'already instrumented through Splunk AO with another tracer provider' RuntimeError for objects that never actually conflicted. This is realistic in any process that creates many short-lived FastAPI apps or TracerProviders (tests, multi-tenant setups, hot-reload dev servers) and is also a slow memory leak since entries are never evicted. Recommend keying by id(obj) only while also holding a weakref to the object (or using WeakValueDictionary/an id->weakref map with a weakref.finalize cleanup callback) so aliasing cannot occur and stale entries are pruned automatically, matching the pattern already used for _configured_providers.
• 🟡 minor (documentation): splunk-ao-migration-tool/README.md (section 6, 'HTTP Tracing Headers') still instructs users to migrate X-Galileo-Trace-ID/X-Galileo-Parent-ID to Splunk-AO-Trace-ID/Splunk-AO-Parent-ID, and says 'The get_tracing_headers() function return value now uses the new header names.' This PR removes Splunk-AO-Trace-ID/Splunk-AO-Parent-ID entirely in favor of W3C traceparent/tracestate, so this guidance is now wrong and will mislead migrating users into propagating headers the SDK no longer understands. Please update this section to point at the W3C headers (or get_tracing_headers()'s new W3C-based output) in the same change, per AGENTS.md's requirement to update docs when propagation/telemetry paths change.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

• src/splunk_ao/decorator.py:1343-1373: _session_id_context (now defined in session_context.py) is a single global ContextVar shared across all SplunkAOLogger instances within a context, rather than being scoped per-logger. _set_active_session_id/set_session_context therefore make one logger's set_session/clear_session call affect get_effective_session_id() for any other logger sharing the same async/thread context (e.g. two loggers for different agent streams created in the same request). This may be intentional per the 'one explicit session, request-local' design, but is worth a design discussion/doc note since it's a behavior change from the previous per-instance self.session_id semantics.
• src/splunk_ao/middleware/tracing.py:16-17: The module docstring's usage example calls logger.conclude(output=str(result)) twice in a row. If unintentional this is a confusing copy-paste artifact in documentation; if intentional (concluding the workflow span then the trace) it deserves a comment explaining why, since readers copying the example verbatim may not realize two distinct steps are being concluded.

[fercor-cisco]

src/splunk_ao/http_instrumentation.py:2441-2443 (line not in diff)

🟠 major (bug): _client_provider_ids and _instrumented_apps key on id(tracer_provider)/id(app) without keeping the objects alive or otherwise validating identity. After the underlying object is garbage collected, Python may reuse the same id for a new, unrelated app/provider, causing silent skip-instrumentation or false ownership-conflict errors for that new object. Track a weakref.ref (or use weakref.finalize to prune the entry when the original object dies) alongside the id, or switch to WeakValueDictionary/WeakSet keyed by the object itself the way _configured_providers already does.

Suggested change

	_client_provider_ids: dict[str, tuple[int, "weakref.ReferenceType[Any]"]] = {}
	_instrumented_apps: WeakSet[Any] = WeakSet() # store (app, provider, framework) via a wrapper holding weakrefs

🤖 Generated by the Astra agent

[fercor-cisco]

🟠 major (bug): Module-level _client_provider_ids: dict[str, int] and _instrumented_apps: set[tuple[int, int, str]] store raw id() values with no reference back to the objects, unlike _configured_providers which is a WeakSet. This is an id-reuse aliasing hazard and an unbounded-growth leak; see PR-level comment for details and a suggested fix (weak references / finalizers keyed alongside the ids).

🤖 Generated by the Astra agent

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-opus-5). It may contain mistakes.

Verdict: request_changes — Several verified defects in new code: the distributed-tracing extra can't actually import its own instrumentors, one bad span aborts the whole handler trace (reproducible from the shipped ADK path), clear_session() cannot clear an inbound baggage session, unfinished callback spans leak OTel context activations, and two flush tests no longer exercise the code they name.

General Comments

• 🟠 major (question): Was _activate_handler_step validated against the async LangChain callback dispatch path?

The new activation mechanism (logger.py:644-665) exists so that the handler span becomes the active W3C parent for outbound HTTP made while a callback is in flight. That requires the otel_context.attach() to take effect in the application's execution context.

But SplunkAOAsyncCallback does not set run_inline, so langchain-core dispatches async callbacks through asyncio.gather(*coros). gather wraps each coroutine in a Task, which copies the current contextvars.Context; mutations inside the Task do not propagate back to the caller. The same applies to sync handlers invoked from ainvoke, which langchain-core runs via run_in_executor(copy_context().run, ...).

If that is right, then for the async paths (a) the attach is invisible to the application, so automatic outbound propagation from inside a handler callback silently does nothing, and (b) the matching detach at end-callback time runs in a different Context, which is what triggers the ERROR-noise problem flagged on logger.py:659. Note _sync_otel_context_impl already anticipates exactly this class of failure (logger.py:496-499) — the new activation path does not.

Exported parent/child topology is unaffected (it comes from _otel_ids, not live context), so this is specifically about the propagation guarantee. Please confirm with a test that drives the real async LangChain callbacks and asserts the injected traceparent parent-id matches the active handler span — the live validation described in the PR body appears to have used @log/openai, not the async framework handlers.

• 🟠 major (testing): The +183 lines of new OpenAI Agents lifecycle code are never driven through the public TracingProcessor interface. grep -rn "on_span_start" tests/ is empty. tests/test_openai_agents.py:46-101 calls _start_owned_root / _start_incremental_span / _finish_incremental_span directly with hand-built Nodes, test_simple_agent uses ingestion_hook (legacy branch), and test_complex_agent is @pytest.mark.skip.

That is why shape mismatches like the output=None case flagged on span_lifecycle.py aren't caught: the tests construct nodes that are known-good rather than nodes the framework actually produces. Please add at least one test that feeds real Span/Trace objects through on_trace_start → on_span_start → on_span_end → on_trace_end on the non-hook path and asserts the exported span set and that _active_steps is empty afterwards.

• 🟡 minor (testing): install_session_propagator() mutates the process-global textmap propagator, and _client_providers / _instrumented_apps / _configured_providers are module-level. These are reset only by the autouse fixture local to tests/test_http_instrumentation.py:119-133; tests/conftest.py has no guard.

Any future test (or an existing one that transitively calls configure_distributed_tracing) outside that file will permanently wrap the global propagator for the rest of the session and pin real TracerProviders in the strong _client_providers dict. Given -n auto xdist and the project's own rule to "reset global OTel context, providers/processors, SDK configuration ... after tests", this belongs in conftest.py as an autouse fixture rather than in one test module.

• 🟡 minor (documentation): The new examples live under examples/logging-samples/DT_2.0/. DT_2.0 is an internal requirement codename that means nothing to a public contributor, which conflicts with AGENTS.md's "Do not document unavailable internal context. Repository documentation must stand alone."

It also duplicates the pre-existing examples/logging-samples/distributed-tracing/, which still ships SPLUNK_AO_MODE=distributed in its .env.example and README — so the repo now contains two DT guides that contradict each other on configuration. Suggest naming the new directories by capability (e.g. distributed-tracing-w3c/ and distributed-tracing-auto/) and either updating or deleting the old sample in this PR, since the PR body's "examples will be fixed in a future PR" note doesn't cover a duplicate that this PR itself creates.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

• src/splunk_ao/handlers/base_async_handler.py:31-95: async_commit/async_end_node duplicate ~55 lines of commit/end_node from the sync base class, differing only in awaiting async_flush. The duplication has already drifted: the _owned_root is not None branch uses serialize_to_str(root_output) for the envelope output (line 66) where the sync path uses SplunkAOLogger._coerce_output(root_output) (base_handler.py:103), so a root output that is a list of ContentBlocks is stringified on the async path and preserved on the sync path. The divergence predates this PR, but this PR copy-pasted the new branch structure into both, doubling the surface. Consider extracting the shared body into a template method taking a flush callable, and aligning the output coercion.
• src/splunk_ao/handlers/span_lifecycle.py:42-48: _step_number is now implemented twice with different behaviour: span_lifecycle._step_number swallows silently, SplunkAOBaseHandler._step_number (base_handler.py:283-292) logs a warning, and log_node_tree (base_handler.py:379-384) inlines a third copy. Consolidate on one helper so the LangGraph step-number rule can't drift between the incremental and legacy paths.
• src/splunk_ao/utils/singleton.py:87-120: Removing trace_id/span_id from the singleton cache key also removed the only per-request differentiation for the decorator path — the deleted docstring said the key existed "for proper isolation of concurrent requests in async web servers." @log on an async endpoint now shares one cached SplunkAOLogger across concurrent requests on the same key. Per-instance ContextVar parent stacks limit the damage, but self.traces and self.session_id remain shared instance state, and the shipped examples/logging-samples/distributed-tracing/retrieval_service.py:56 uses exactly that pattern. Worth a design note or a request-scoped key that doesn't depend on the removed proprietary IDs.
• src/splunk_ao/handlers/openai_agents/handler.py:60-64: SplunkAOTracingProcessor is installed process-wide but keeps single-valued per-trace state (_owned_trace, _caller_parent, _owned_root, _owned_root_node_id, _nodes, _active_steps). Partly pre-existing for _nodes/_owned_trace, but this PR adds three more fields with the same shape. Keying all per-trace state by trace_id would make concurrent Runner.run(...) calls safe and is a prerequisite for the on_trace_end cleanup fix flagged inline.

[pradystar]

🤖 This review was generated by the Astra agent (claude-opus-5). It may contain mistakes.

Verdict: request_changes — Several verified defects in new code: the distributed-tracing extra can't actually import its own instrumentors, one bad span aborts the whole handler trace (reproducible from the shipped ADK path), clear_session() cannot clear an inbound baggage session, unfinished callback spans leak OTel context activations, and two flush tests no longer exercise the code they name.

General Comments

• 🟠 major (question): Was _activate_handler_step validated against the async LangChain callback dispatch path?

The new activation mechanism (logger.py:644-665) exists so that the handler span becomes the active W3C parent for outbound HTTP made while a callback is in flight. That requires the otel_context.attach() to take effect in the application's execution context.

But SplunkAOAsyncCallback does not set run_inline, so langchain-core dispatches async callbacks through asyncio.gather(*coros). gather wraps each coroutine in a Task, which copies the current contextvars.Context; mutations inside the Task do not propagate back to the caller. The same applies to sync handlers invoked from ainvoke, which langchain-core runs via run_in_executor(copy_context().run, ...).

If that is right, then for the async paths (a) the attach is invisible to the application, so automatic outbound propagation from inside a handler callback silently does nothing, and (b) the matching detach at end-callback time runs in a different Context, which is what triggers the ERROR-noise problem flagged on logger.py:659. Note _sync_otel_context_impl already anticipates exactly this class of failure (logger.py:496-499) — the new activation path does not.

Exported parent/child topology is unaffected (it comes from _otel_ids, not live context), so this is specifically about the propagation guarantee. Please confirm with a test that drives the real async LangChain callbacks and asserts the injected traceparent parent-id matches the active handler span — the live validation described in the PR body appears to have used @log/openai, not the async framework handlers.

• 🟠 major (testing): The +183 lines of new OpenAI Agents lifecycle code are never driven through the public TracingProcessor interface. grep -rn "on_span_start" tests/ is empty. tests/test_openai_agents.py:46-101 calls _start_owned_root / _start_incremental_span / _finish_incremental_span directly with hand-built Nodes, test_simple_agent uses ingestion_hook (legacy branch), and test_complex_agent is @pytest.mark.skip.

That is why shape mismatches like the output=None case flagged on span_lifecycle.py aren't caught: the tests construct nodes that are known-good rather than nodes the framework actually produces. Please add at least one test that feeds real Span/Trace objects through on_trace_start → on_span_start → on_span_end → on_trace_end on the non-hook path and asserts the exported span set and that _active_steps is empty afterwards.

• 🟡 minor (testing): install_session_propagator() mutates the process-global textmap propagator, and _client_providers / _instrumented_apps / _configured_providers are module-level. These are reset only by the autouse fixture local to tests/test_http_instrumentation.py:119-133; tests/conftest.py has no guard.

Any future test (or an existing one that transitively calls configure_distributed_tracing) outside that file will permanently wrap the global propagator for the rest of the session and pin real TracerProviders in the strong _client_providers dict. Given -n auto xdist and the project's own rule to "reset global OTel context, providers/processors, SDK configuration ... after tests", this belongs in conftest.py as an autouse fixture rather than in one test module.

• 🟡 minor (documentation): The new examples live under examples/logging-samples/DT_2.0/. DT_2.0 is an internal requirement codename that means nothing to a public contributor, which conflicts with AGENTS.md's "Do not document unavailable internal context. Repository documentation must stand alone."

It also duplicates the pre-existing examples/logging-samples/distributed-tracing/, which still ships SPLUNK_AO_MODE=distributed in its .env.example and README — so the repo now contains two DT guides that contradict each other on configuration. Suggest naming the new directories by capability (e.g. distributed-tracing-w3c/ and distributed-tracing-auto/) and either updating or deleting the old sample in this PR, since the PR body's "examples will be fixed in a future PR" note doesn't cover a duplicate that this PR itself creates.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

• src/splunk_ao/handlers/base_async_handler.py:31-95: async_commit/async_end_node duplicate ~55 lines of commit/end_node from the sync base class, differing only in awaiting async_flush. The duplication has already drifted: the _owned_root is not None branch uses serialize_to_str(root_output) for the envelope output (line 66) where the sync path uses SplunkAOLogger._coerce_output(root_output) (base_handler.py:103), so a root output that is a list of ContentBlocks is stringified on the async path and preserved on the sync path. The divergence predates this PR, but this PR copy-pasted the new branch structure into both, doubling the surface. Consider extracting the shared body into a template method taking a flush callable, and aligning the output coercion.
• src/splunk_ao/handlers/span_lifecycle.py:42-48: _step_number is now implemented twice with different behaviour: span_lifecycle._step_number swallows silently, SplunkAOBaseHandler._step_number (base_handler.py:283-292) logs a warning, and log_node_tree (base_handler.py:379-384) inlines a third copy. Consolidate on one helper so the LangGraph step-number rule can't drift between the incremental and legacy paths.
• src/splunk_ao/utils/singleton.py:87-120: Removing trace_id/span_id from the singleton cache key also removed the only per-request differentiation for the decorator path — the deleted docstring said the key existed "for proper isolation of concurrent requests in async web servers." @log on an async endpoint now shares one cached SplunkAOLogger across concurrent requests on the same key. Per-instance ContextVar parent stacks limit the damage, but self.traces and self.session_id remain shared instance state, and the shipped examples/logging-samples/distributed-tracing/retrieval_service.py:56 uses exactly that pattern. Worth a design note or a request-scoped key that doesn't depend on the removed proprietary IDs.
• src/splunk_ao/handlers/openai_agents/handler.py:60-64: SplunkAOTracingProcessor is installed process-wide but keeps single-valued per-trace state (_owned_trace, _caller_parent, _owned_root, _owned_root_node_id, _nodes, _active_steps). Partly pre-existing for _nodes/_owned_trace, but this PR adds three more fields with the same shape. Keying all per-trace state by trace_id would make concurrent Runner.run(...) calls safe and is a prerequisite for the on_trace_end cleanup fix flagged inline.

• major: async LangChain activation
  Fixed. Both synchronous and asynchronous LangChain callbacks now setrun_inline=True, so handler activation occurs in the application’s executioncontext instead of an isolated task or executor context. A regression testdrives the real async callback manager and verifies that the injectedtraceparent uses the active handler span as its parent.
• major: OpenAI Agents public lifecycle testing
  Fixed. The tests now exercise the public OpenAI Agents tracing lifecycle ratherthan only calling private helper methods. Coverage drives trace and span start/end callbacks, asserts the exported span set and parent relationships, verifiescleanup, and includes concurrent traces to prove state isolation.
• minor: global test-state fixture
  No product change was needed here. The HTTP instrumentation tests have anautouse fixture that restores the previous global propagator and clears the SDKbookkeeping after every test. Moving that guard into the repository-wideconftest.py would be test-infrastructure centralization rather than acorrectness fix, so it is not required for this change.
• minor: DT_2.0 and conflicting examples
  Partially accepted. The pre-existing distributed-tracing sample was updated toremove the stale SPLUNK_AO_MODE=distributed guidance and align its environmentvariables and README with the current behavior, so the two examples no longercontradict each other. The DT_2.0 directory name is intentionally retained asthe requested example grouping.

———

• Follow-up#1: sync/async handler duplication
  Agreed as follow-up cleanup. The sync/async duplication and output-coerciondifference predate this change and should be addressed together through a sharedlifecycle template rather than modified incidentally here. This change keeps theexisting sync and async behavior intact while testing the new lifecycle behaviordirectly.
• Follow-up#2: duplicate _step_number logic
  Agreed as follow-up cleanup. Consolidating the LangGraph step-number extractioninto one helper would reduce drift, but it is not required for the propagationor incremental batching behavior implemented here.
• Follow-up#3: request-scoped singleton key
  I agree that concurrent logger state deserves explicit design scrutiny, but I donot recommend restoring trace/span IDs to the singleton key. That would createrequest-scoped logger instances, exporters, BatchSpanProcessor threads, andshutdown hooks. Execution-local parent and session selection now use context-local state; any remaining shared mutable state should be isolated at that layerin a dedicated concurrency follow-up rather than by multiplying loggerinstances.
• Follow-up#4: OpenAI Agents process-wide scalar state
  Fixed in this change because it directly affected the new lifecycleimplementation. OpenAI Agents state is now maintained per trace ID instead of inprocessor-wide scalar fields, guarded during lifecycle updates, and removed wheneach trace completes. Concurrent public trace tests verify that one trace cannotclear or overwrite another trace’s nodes or active steps.