You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(eventhubs): add tracing and AMQP lifecycle observability - #7368
Event Hubs send and receive operations now emit provider-neutral tracing through Azure Core. Producer and consumer options expose TracingProvider; logical operation spans cover retries, and child spans measure AMQP link attachment, send disposition, and blocking receive latency. Structured lifecycle logs identify connections, sessions, links, and their generations. The runtime package has no direct OpenTelemetry dependency and emits no metrics directly; applications can derive counters and histograms from lifecycle records and spans.
Receiver recovery preserves the identity of the receiver being closed and allocates a new generation for each replacement attempt:
active receiver 1 -> close failure: generation 1
replacement 2 -> attach failure: generation 2
replacement 3 -> attach failure: generation 3
replacement 4 -> attach succeeds: active generation 4
Receiver instances have distinct diagnostic identities across partition-client recreation. Pending-error replay avoids duplicate lifecycle records, and peer-detached senders emit structured discard records. Event Hubs waits for confirmed uAMQP receiver attachment before ending attachment spans or reporting success; rejected, cancelled, and timed-out opens release their resources. The default AMQP receiver open remains asynchronous. Windows x86 builds use a 64-bit Rust host toolchain with the 32-bit target.
An opt-in OpenTelemetry sample provides runnable OTLP/HTTP trace and log exporters, an Azure Logger bridge, local collector configuration, build/run instructions, and provider shutdown. A no-Azure smoke mode verifies export; a separate live send mode demonstrates SDK tracing. Documentation distinguishes unsupported native SDK metrics from application-owned and backend-derived metrics.
Rebased onto main addf6289b; the final rebase only reconciled release-note headings in the affected packages. Sample and observability source are unchanged from the validated revisions. Preserved CBS retry jitter and sender liveness checks. Release notes and package versions target Event Hubs 1.0.0-beta.17 and AMQP 1.0.0-beta.15 (unreleased).
uAMQP: 101/101 focused Event Hubs tests and 183/183 core AMQP tests passed. Rust: 100/100 focused Event Hubs tests and 177/177 core AMQP tests passed. Core selection excludes TLS transport tests; Event Hubs selection requires no Azure service credentials.
Five receiver regressions cover delayed successful attachment, null-source ATTACH followed by rejection DETACH, cancellation/deadline cleanup, and default asynchronous behavior. The original implementation failed the four opt-in waiting cases; the default case passed. The rejection test also failed before the narrow native null-source guard.
The real peer-detach producer regression emitted zero structured discard records before the fix and the expected single Informational record afterward. Three receiver identity tests cover distinct instance IDs, recovery continuity, and shared connection/session failure scopes.
Pending-error replay and the producer stack-ID race were checked against the implementation; no new end-to-end replay or deterministic concurrency test was added.
Local macOS uAMQP loopback tests used the same pinned c-shared-utility source rebuilt with its supported OpenSSL/Berkeley socket configuration because the installed dependency's socket adapter was a stub. This was a temporary local build override; the shared dependency installation and repository manifests were unchanged.
Independent review found no blockers. Changed-line clang-format 20, cspell, and whitespace checks pass. Windows/Linux execution remains for CI; no live Azure service tests were run for these fixes.
Earlier validation:
Exporter sample (4e28389): built on uAMQP and Rust AMQP with the pinned opentelemetry-cpp 1.28.0 dependencies. Both smoke runs exported sample.exporter-check and actual SDK logs to a local OpenTelemetry Collector 0.109.0. Collector output verified the service resource and INFO/DEBUG severity mapping. The smoke span is application-generated; live Azure send/receive was not exercised.
The exporter sample is opt-in. No runtime library or root dependency-manifest changes were made for it. Clang-format 20.1.8 and whitespace checks pass.
Runtime validation at d7cf738b6 (runtime code unchanged by the sample commit):
Before: after extracting the existing recovery lifecycle into the private routine used by PartitionClient, both new regression tests failed: close failure used generation 8 instead of 7, and three attachment attempts reused generation 2.
After: close failures retain the active generation, failed attempts consume distinct generations, and cancellation during close stops recovery without allocating a replacement. All three regression tests pass on both AMQP backends.
macOS ARM64, Apple Clang 21, C++14, Debug: Event Hubs test executable and all eight public samples built on uAMQP and Rust AMQP.
97/97 offline tests passed on each backend, using AZURE_TEST_MODE=PLAYBACK: tracing, connection-string construction, event data/batches, processor load balancing/close, reattachment policy, and retry operations. This is a focused selection, not the complete service suite.
Local builds retain warnings-as-errors except for two existing compiler diagnostics: C++ unnecessary-virtual-specifier and vendored C comment, configured with -Wno-error for those diagnostics only.
Changed-line clang-format 20.1.8 and git diff --check pass.
The earlier PR revision passed a live Azure send/receive smoke test with the OpenTelemetry ostream exporter and emitted 17 lifecycle records. That live test was not repeated after this rebase; Windows/Linux and current platform-matrix validation are delegated to CI.
Merge Danger
Door: two-way
Blast Radius: observability and uAMQP receiver attachment
Changes span lifetimes, diagnostic identities, and failure reporting in Event Hubs and uAMQP. The public TracingProvider fields and EventDataBatch::NumberOfEvents() const signature require maintainer/API review before merge; approval has not been established. Fresh CI must be evaluated after the review fixes. The internal receiver wait and the narrow vendored null-source attachment guard change uAMQP receiver behavior and require platform-matrix validation. Earlier documentation checks passed after correction; subsequent runs also encountered GitHub checkout/download failures and the Windows x86 Rust host-toolchain removal, for which this branch now includes a build fix. End-to-end message-context propagation, batch links, and direct metrics remain outside this change.
Azure Pipelines:
Successfully started running 3 pipeline(s).
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The empty Breaking Changes section fails changelog validation, while unresolved tracing and receiver-lifecycle defects produce inaccurate spans, reused receiver identities, and duplicate error records.
No open issues 🎉
8 previously missed issues
In code that hasn't changed since last review
Move tracing changelog entries to unreleased versions
The tracing entry is added under Event Hubs beta.14, which was already published without this feature, so it assigns tracing to the wrong release. Move it into a new unreleased section and advance the package version rather than editing released history. The same issue affects sdk/core/azure-core-amqp/CHANGELOG.md:37, since beta.13 has also shipped.
The OpenTelemetry adapter makes its sampling decision inside CreateTracingContext, before the messaging attributes are added here. A sampler configured to retain Event Hubs or a specific destination can therefore discard matching operations because it cannot see those attributes at span creation. Populate createOptions.Attributes with all known messaging and AMQP attributes before creating the span; add only completion-time values later.
Allocate unique generations for every receiver creation attempt
Every new partition client for the same consumer and partition restarts at generation 1, so replacement receivers reuse component IDs. RebuildReceiver also reuses a generation after an attach throws, because it advances only on success (partition_client.cpp:330,370). Allocate receiver generations from shared per-partition state for every creation attempt, including failed attachments, and retain the active generation separately for close/failure records.
Wait for receiver attach completion before reporting success
uAMQP's MessageReceiverImpl::Open returns before remote attachment completes, so this reports success before peer acceptance and omits handshake latency. Wait for the OPEN/ERROR result before ending the span, and emit attached only for OPEN. Same issue in RebuildReceiver at partition_client.cpp:368-371.
Use the active receiver generation for close failures
The close that just failed operated on m_receiverGeneration, but diagnosticsContext identifies the replacement generation. Closing generation 1 therefore records close_failed against generation 2 before that receiver exists. Build this failure context from the current receiver generation and reserve the replacement context for attachment records.
Avoid duplicate failure records for pending-error replay
When a receive returns buffered events with m_pendingError, the next call passes that same error to recover and emits another failed record. The original observation or failed reattach already logged it, so counters derived from lifecycle records count the same fault twice. Skip this lifecycle record for pending-error replay, or emit it only where a new transport error is observed.
A context cancelled before entry, or between iterations, exits this loop without throwing, so the receive span records neither cancellation nor error status. Record a cancellation exception on the span when cancellation stops the loop, while preserving the existing empty or partial return behavior.
This factory logs each receive error at Error level, then recover(result.second) invokes it again for the same error. That doubles identical error logs even when tracing is disabled, inflating error-based alerts and counters. Guard the span annotation with HasTracer() and create its exception without the logging factory.
Add eventhubs_tracing_test.cpp with the recording tracer, span, and
attribute set doubles for the Event Hubs tracing work in issue 7336.
The tests pin the TracingProvider option on ProducerClientOptions and
ConsumerClientOptions, the eager tracer creation in both clients, the
ProducerClient.Send span shape on a cancelled context and on a
non-cancel exception, and the shape of the shared StartSpan helper for
PartitionClient.ReceiveEvents.
Tests 1 to 7 do not compile until the tracing source lands. The
TracingProvider member does not exist yet, and the private header
src/private/eventhubs_tracing.hpp does not exist yet.
SendWithoutProviderIsUnchanged is a characterization test. It passes
today. It guards the no-provider path against a null dereference after
the source lands.
ProducerClientOptions and ConsumerClientOptions gain a TracingProvider
field. Each client builds a TracingContextFactory from that field at
construction time. When the field is empty, the factory has no tracer
and the clients create no spans.
ProducerClient::Send creates one Producer span named
ProducerClient.Send. PartitionClient::ReceiveEvents creates one Client
span named PartitionClient.ReceiveEvents. Both spans carry the
messaging attributes, the az.namespace attribute, and the message
count. Both spans record an exception event and an error status, then
rethrow the original exception.
A new private helper, src/private/eventhubs_tracing.hpp, holds the
factory creation and the span shape, so the producer and the partition
client emit the same attributes.
EventDataBatch::NumberOfEvents is now const, and its mutex is mutable,
because Send takes the batch by const reference.
Trace-context propagation and span links stay deferred, because Azure
Core has no API for them.
Run clang-format-11 over eventhubs_tracing_test.cpp. The declarations
of AddAttributes and SetStatus on the recording span double broke the
Validate Clang Format step in CI.
The change is formatting only. No assertion, no test name, and no
behavior changed.
ProducerClient::Send converted the batch to an AMQP message before it
started the span. An empty batch makes that conversion throw, so the
send failed before a span existed and nothing recorded the failure.
The conversion now runs at the top of the try block, after the span
starts. The existing catch clause adds the exception to the span and
rethrows, the same as every other send failure.
The OpenTelemetry semantic conventions at schema 1.17.0 define
messaging.batch.message_count as an int. The Event Hubs tracing helper
records it as a string, and the test doubles could not show the
difference, because they stringified every overload into one map.
TestAttributeSet and TestSpan now record a type tag for each attribute.
The tag names the overload that delivered the value. Three tests use it:
the receive span count and the send span count must arrive through the
uint64 overload, and SetMessageCount must stay quiet and safe when the
factory has no tracer. The first two tests are red today. The third one
is a characterization test that guards the null attribute set hazard.
The batch message count now goes out as an unsigned integer attribute,
which matches the OpenTelemetry semantic conventions at schema 1.17.0.
Before this change the helper wrote the count as a string.
SetMessageCount takes the tracing context factory, because the typed
attribute needs an attribute set. The helper guards the null attribute
set on the no-tracer path, because the factory returns nullptr when it
has no tracer and that pointer is not null safe.
The span helper built a std::string temporary for each attribute name
before it added the attribute. Two of the names are longer than the
libc++ small-string limit, so each call heap allocated even when the
client had no tracer and dropped every attribute.
The span helper now returns the tracing context immediately when the
factory has no tracer. The message count helper keeps its own null
attribute set guard, because PartitionClient::ReceiveEvents calls it
directly.
The remaining two allocations per operation come from azure-core and
are tracked in issue 7340.
The SingleSpan helper called front() on the tracer list after a
non-fatal EXPECT_EQ on the list size. When no tracer was created, that
call dereferenced an empty std::list and the test process ended in a
segmentation fault. The crash stopped the run, so the tests after the
first affected one never reported at all.
The helper now returns null when the tracer list is not the expected
size, so a failure reports through the caller's ASSERT_NE assertion
rather than a crash. This mirrors the guard that already protects the
spans list two lines later.
The receive span of PartitionClient::ReceiveEvents has no offline
coverage. A partition client comes only from a factory that opens a
live AMQP link, so no unit test can reach the span, the message count,
or the exception path.
This test runs against a real namespace. It gives the consumer client
a recording tracing provider, receives events from a partition, and
makes sure the span name, the span kind, the messaging attributes, and
the message count are correct. The count assertion compares the span
attribute with the size of the vector that ReceiveEvents returned.
The recording tracing test doubles move from eventhubs_tracing_test.cpp
into a shared header, because the live test lives in a different file.
The two convenience Send overloads build the batch before they send it,
and CreateBatch opens the connection, the session and the AMQP link. A
bad host fails there. Only the batch overload starts a span today, so
that failure unwinds with no span at all.
Add two offline tests that send one event and a vector of three events
against a fake connection string. Each test asserts one span named
ProducerClient.Send with the kind Producer, the message count as a
uint64 attribute, one recorded event and the Error status. Both fail
today because the provider holds zero spans.
Add a characterization test that keeps the no-provider path unchanged,
and a live test that sends one event and asserts a single span. The
live test catches a nested second span, which an offline test cannot
reach because CreateBatch always throws offline.
The convenience Send overloads made the batch first, and the batch
creation opens the connection, the session, and the AMQP link. A bad
host or a failed authentication threw there, and the stack unwound
before a span existed. A caller then saw no producer span for the
failed send.
Each public Send overload now starts one span, and one try/catch
records the exception on that span. A new private method,
SendBatchInSpan, holds the send body and starts no span, so one
logical send still makes one span. The convenience overloads pass the
span context to CreateBatch, which puts the link open under the span.
The tracing change added a TracingProvider field to
ProducerClientOptions and ConsumerClientOptions, but no document told
a user how to set it. The only repo document on tracing,
doc/DistributedTracing.md, showed clientOptions.Telemetry.TracingProvider.
That line does not compile for Event Hubs, because the two option
structs do not derive from Azure::Core::_internal::ClientOptions and
have no Telemetry field.
- Adds a "Distributed tracing" section to the Event Hubs README with a
snippet that builds an OpenTelemetryProvider and sets TracingProvider
on both client options.
- Lists the span names, the span kinds, and the attributes that the
clients emit, and names the 1.17.0 semantic conventions schema.
- States that the client creates no spans when TracingProvider is not
set, because there is no global fallback provider.
- Adds a note to doc/DistributedTracing.md for the clients that declare
TracingProvider at the top level of their own options structure.
Identify Event Hubs AMQP connections, sessions, and links in lifecycle logs and record their failures, teardown, and recreation generations. Attribute scoped AMQP conditions to the failed protocol layer.
Add child spans around link attachment, send disposition, and blocking receive calls so exporters can separate caller-visible SDK latency from AMQP transport latency. Document how to derive failure counters and latency histograms from logs and spans.
Anton Kolesnyk (@antkmsft), this PR is still active and needed for Event Hubs tracing and AMQP lifecycle diagnostics. I rebased it, added exporter documentation and a runnable sample, and pushed fixes for the six review threads in a5686d0ec. GitHub currently reports no merge conflicts. The focused local checks passed on both AMQP backends; fresh CI and maintainer/API review remain pending.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Event Hubs send and receive operations now emit provider-neutral tracing through Azure Core. Producer and consumer options expose
TracingProvider; logical operation spans cover retries, and child spans measure AMQP link attachment, send disposition, and blocking receive latency. Structured lifecycle logs identify connections, sessions, links, and their generations. The runtime package has no direct OpenTelemetry dependency and emits no metrics directly; applications can derive counters and histograms from lifecycle records and spans.Receiver recovery preserves the identity of the receiver being closed and allocates a new generation for each replacement attempt:
Receiver instances have distinct diagnostic identities across partition-client recreation. Pending-error replay avoids duplicate lifecycle records, and peer-detached senders emit structured discard records. Event Hubs waits for confirmed uAMQP receiver attachment before ending attachment spans or reporting success; rejected, cancelled, and timed-out opens release their resources. The default AMQP receiver open remains asynchronous. Windows x86 builds use a 64-bit Rust host toolchain with the 32-bit target.
An opt-in OpenTelemetry sample provides runnable OTLP/HTTP trace and log exporters, an Azure Logger bridge, local collector configuration, build/run instructions, and provider shutdown. A no-Azure smoke mode verifies export; a separate live send mode demonstrates SDK tracing. Documentation distinguishes unsupported native SDK metrics from application-owned and backend-derived metrics.
Rebased onto main
addf6289b; the final rebase only reconciled release-note headings in the affected packages. Sample and observability source are unchanged from the validated revisions. Preserved CBS retry jitter and sender liveness checks. Release notes and package versions target Event Hubs1.0.0-beta.17and AMQP1.0.0-beta.15(unreleased).Supersedes #7339. Fixes #7336.
Evidence
Review-fix validation at
a5686d0ec:Earlier validation:
sample.exporter-checkand actual SDK logs to a local OpenTelemetry Collector 0.109.0. Collector output verified the service resource and INFO/DEBUG severity mapping. The smoke span is application-generated; live Azure send/receive was not exercised.Runtime validation at
d7cf738b6(runtime code unchanged by the sample commit):PartitionClient, both new regression tests failed: close failure used generation 8 instead of 7, and three attachment attempts reused generation 2.AZURE_TEST_MODE=PLAYBACK: tracing, connection-string construction, event data/batches, processor load balancing/close, reattachment policy, and retry operations. This is a focused selection, not the complete service suite.unnecessary-virtual-specifierand vendored Ccomment, configured with-Wno-errorfor those diagnostics only.git diff --checkpass.Merge Danger
Door: two-way
Blast Radius: observability and uAMQP receiver attachment
Changes span lifetimes, diagnostic identities, and failure reporting in Event Hubs and uAMQP. The public
TracingProviderfields andEventDataBatch::NumberOfEvents() constsignature require maintainer/API review before merge; approval has not been established. Fresh CI must be evaluated after the review fixes. The internal receiver wait and the narrow vendored null-source attachment guard change uAMQP receiver behavior and require platform-matrix validation. Earlier documentation checks passed after correction; subsequent runs also encountered GitHub checkout/download failures and the Windows x86 Rust host-toolchain removal, for which this branch now includes a build fix. End-to-end message-context propagation, batch links, and direct metrics remain outside this change.