fix(otel): send lazily resolved API key with opentelemetry-exporter-otlp-proto-http 1.45 - #811
Conversation
…tlp-proto-http 1.45 opentelemetry-exporter-otlp-proto-http 1.45.0 moved request headers off the exporter (`_headers` / `_session`) and onto an internal `_client`. When the Braintrust API key was resolved after the exporter was constructed (e.g. from `.env.braintrust` or an env var set later), `OtelExporter._set_api_key_header` updated attributes the exporter no longer reads, so spans were exported without an `Authorization` header. Instead of patching upstream's private header storage, re-run the public `OTLPSpanExporter.__init__` with the resolved `Authorization` header. Track `shutdown()` so resolving a key cannot revive an exporter that was already shut down. Replace the tests that asserted on private exporter state with tests that export to a local OTLP collector (the shared `scripted_server` helper) and check the headers it receives. Add an OpenTelemetry version matrix for `test_otel` (latest=1.45.0, 1.44.0, 1.16.0) so both the pre- and post-1.45 exporter layouts stay covered. 1.16.0 is the oldest release that imports without pkg_resources; it is skipped on Python 3.14 because OpenTelemetry <1.28 requires protobuf<5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65f23c1079
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| if "Authorization" not in exporter_kwargs["headers"] and not self._braintrust_shutdown: | ||
| # Re-run the upstream constructor instead of patching its private header | ||
| # storage, which moved in opentelemetry-exporter-otlp-proto-http 1.45. | ||
| headers = {"Authorization": f"Bearer {api_key}", **exporter_kwargs["headers"]} | ||
| super().__init__(**{**exporter_kwargs, "headers": headers}) |
There was a problem hiding this comment.
When the first lazy-key export races with shutdown()—for example, a SimpleSpanProcessor ending a span on one thread while its provider shuts down on another—the shutdown flag can change after this check but before super().__init__(). The constructor then resets the upstream shutdown state and recreates its HTTP client after shutdown, so the current and subsequent exports can continue using an exporter that was supposed to remain closed. Protect the check/reinitialization and shutdown transition with the same lock.
Useful? React with 👍 / 👎.
When the first lazy-key export raced with shutdown() (e.g. a SimpleSpanProcessor ending a span on one thread while the provider shuts down on another), shutdown could land between the shutdown check and the upstream re-init. Re-running OTLPSpanExporter.__init__ then reset the upstream shutdown state and recreated the HTTP client, so the exporter kept sending after it had been shut down. Guard the check/re-init and the shutdown transition with one lock, and re-check the resolved-key latch under it so concurrent first exports only re-init once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
Summary
opentelemetry-exporter-otlp-proto-http1.45.0 (released 2026-09-25) moved the exporter's request headers off_headers/_sessionand onto an internal HTTP client. WhenOtelExporterresolved the Braintrust API key after construction, for example from.env.braintrustor an env var set later, it wrote theAuthorizationheader to attributes the exporter no longer reads. Spans were exported without authentication. The key-at-construction path still worked.test_otelinstalls otel unpinned, so it started failing in CI on every PR (e.g. #809, shard 2 on all Pythons and OSes).1. Fix
OtelExporterstores the kwargs it passes toOTLPSpanExporter.__init__. When the key is resolved lazily, it re-runs that public constructor with theAuthorizationheader added, instead of writing into private upstream state. The constructor is the only public way to set headers across 1.16–1.45.OtelExporter.shutdown()now records the shutdown, so resolving a key can't revive an exporter that was already shut down. Re-running the constructor resets upstream's shutdown flag.Authorizationheader still takes precedence, and.env.braintrustis still only read on first export.2. Tests
exporter._headersnow export a span to a local OTLP collector (the sharedbraintrust.api._test_server.scripted_server) and assert on the headers it receives. The old.env.braintrusttest passed on 1.45 while no header was actually sent.Authorizationheader; resolving a key doesn't revive a shut-down exporter.3. OpenTelemetry nox matrix
test_otelis now parametrized overlatest(1.45.0), 1.44.0 (the last version before the change) and 1.16.0, pinned in[tool.braintrust.matrix]as separateopentelemetry-api/-sdk/-exporter-otlp-proto-httptables. They are kept separate because older exporters pin the SDK only loosely, e.g. 1.12.0 acceptsopentelemetry-sdk~=1.11.pkg_resources.protobuf<5, which crashes on 3.14, sotest_otelskips them there. 1.44.0 still covers the pre-1.45 layout on 3.14.session-weights.jsongets the parametrized session names.update-matrix-latest.pywill bump these pins. It classifies them as provider packages, so otel bumps are labeledneeds-cassette-rerecordfor human review and are not auto-merged.Test plan
nox -s "test_otel(latest)","test_otel(1.44.0)"and"test_otel(1.16.0)"pass on Python 3.10 and 3.13. On 3.14,latestand 1.44.0 pass and 1.16.0 is skipped.test_otel_not_installedpasses. Thesrc/braintrust/otel/tests (which CI only runs without otel installed) pass on every matrix version.Authorization: Bearer …to a local HTTP server on 1.16.0, 1.44.0 and 1.45.0. Before the fix, 1.45.0 sent noAuthorizationheader.pre-commitandnox -s pylintNotes
🤖 Generated with Claude Code
Co-authored by StarfolkAI (@starfolkai)[bot]