path based routing for signals causing issue with /v1/metrics #48194
kadhamecha-conga
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi Team,
I’m experimenting with FQDN endpoints and path-based routing via NGINX for OpenTelemetry signals (/v1/traces, /v1/metrics, /v1/logs). Each path is routed to separate backend pods. centralized otel collector uses prometheusremotewrite to push data in mimir.
Traces and logs are working as expected, but I’m encountering an issue with metrics:
Metrics requests are failing with HTTP 500 errors
Errors are marked as non-retryable: “Exporting failed. The error is not retryable. Dropping data.”
OTEL clients (especially DaemonSets) start dropping metrics after retry attempts
Has anyone seen this behavior or have suggestions on what might be causing it?
All reactions