[exporter/loadbalancing] retry_on_failure replays the whole fan-out; return partial-data errors so only failed endpoints' data is retried

[AirbornePorcine]

Component(s)

exporter/loadbalancing

Is your feature request related to a problem? Please describe.

#36094 added top-level sending_queue and retry_on_failure to this exporter, and the README recommends them for elastic environments so data can be "re-routed into a new set of healthy backends". At fan-out width, the retry does much more than re-route: it duplicates.

ConsumeTraces splits an incoming request across N backend endpoints and joins per-endpoint errors with multierr. When any one endpoint fails, exporterhelper's retry re-invokes the exporter with the whole original request, which re-enters ConsumeTraces, re-hashes everything on the current ring, and re-sends to every endpoint — including the ones that already accepted their share. Duplication therefore scales with fan-out width, and endpoint failure at scale is not rare: it happens on every scale-in, node consolidation, and spot reclaim.

Measured in our production clusters (a loadbalancer tier fanning out to on the order of a hundred backend endpoints, with continuous node churn), with retry_on_failure enabled on the loadbalancing exporters:

• Our egress-to-external-ingress ratio nearly doubled (a ~74% egress inflation) while external ingress stayed flat — the extra spans were duplicates reaching our backend.
• Retries never converged, because at continuous churn there is always a recently-removed endpoint; the failed-send rate stayed elevated for the whole window.
• We reverted the configuration within the hour.

A narrow environment cannot see this: our staging clusters (3 endpoints per tier) showed rejects matched 1:1 by retries and steady delivery. The failure mode only appears at width.

Describe the solution you'd like

Return partial-data errors from the fan-out: collect the items belonging to the endpoints that failed and return consumererror.NewTraces(joinedErr, failedOnly) (and the metrics/logs equivalents), instead of a plain joined error.

The machinery for this already exists and composes today:

• exporterhelper's retry sender calls request.OnError(err) before each backoff, and consumererror.Traces implements the extraction — so the retried request becomes only the failed subset, re-hashed on the current ring (which is exactly the re-route the README promises).
• Endpoints that accepted their share are never re-sent. Residual duplication shrinks to send-failed-after-partial-write cases — bounded per failure instead of fan-out-wide.

The change is local to the ConsumeTraces/ConsumeMetrics/ConsumeLogs fan-out in this component; no exporterhelper changes are required.

One thing the PR must pin down with a test: what otelcol_exporter_send_failed_spans records when a partial retry exhausts (the residual failed subset, or the original request size). The observation wrapper sits outside the retry loop, and this counter's semantics matter to anyone alerting on the component.

We intend to submit the PR.

Describe alternatives you've considered

• Leaving retry off (status quo for us): endpoint removal silently loses the removed endpoint's in-flight share — roughly 1.5% of trace volume at our churn rate, invisible to ratio-based alerting.
• Membership fixes (kubernetes resolver condition filtering — [exporter/loadbalancing] k8s resolver ignores endpoint readiness and keeps routing new data to terminating backends #50436; dns resolver + headless Service; preStop drains): they reduce how often an endpoint fails mid-send, but none covers forceful terminations (spot reclaims, OOM kills). A retry that replays only failures composes with all of them.
• Per-endpoint child queues (queue under protocol.otlp): the child queue acknowledges the batch, then is deleted together with the removed endpoint's exporter (removeExtraExporters shuts it down immediately), so it cannot re-route what it holds.

Additional context

• Duplicate spans generated by retries opentelemetry-collector#11056 tracks generic at-least-once duplication for single-hop retries; this request is specifically about fan-out amplification in this component, which turns one failed endpoint into an N-endpoint replay.
• In [exporter/loadbalancing] Support consistency between scale-out events #33959 a code owner acknowledged the scale-in flush gap this would close from the retry side.
• Behaviour verified against v0.153.0 and current main.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @rlankfo @iblancasa

See Adding Labels via Comments if you do not have permissions to add labels yourself.