How does the prometheusremotewrite exporter handle k8s services as its destination?

[ronnybremer]

When deploying the prometheus-stack via helm in a kubernetes cluster, two instances of Prometheus are created by default. A single service exposes them, so I can easily use:

exporters:
    prometheusremotewrite:
      endpoint: "http://kube-prom-stack-kube-prome-prometheus:9090/api/v1/write"
      tls:
        insecure: true
      resource_to_telemetry_conversion:
        enabled: true

in the otel-collector config.

However, this performs a load-balancing between both instances and I assume, metrics only go to a single instance (more or less randomly in one or the other).
If my assumption is correct, I could create a headless service, exposing both IP addresses when queried via DNS.

How does the exporter handle this scenario, does it export the metrics to all returned IP addresses or only to one of them?

Thank you for your input.

[s3onghyun]

Your assumption is correct, and a headless service will not fix it.

A ClusterIP Service load-balances per connection, so the collector's remote-write requests are distributed across the two Prometheus replicas: each replica ends up with a partial, non-overlapping slice of your data. Every query then returns whichever fraction happened to land on the replica it hit — which is worse than it looks, because nothing errors.

A headless service does not change the outcome. It just returns all pod IPs in DNS instead of one virtual IP; the HTTP client still picks one endpoint per request and writes there. You would go from "load balanced by kube-proxy" to "load balanced by DNS/dialer" — still one destination per request, still split data.

What the prometheus-stack HA pair actually expects is that both replicas receive the full stream independently, and you deduplicate at query time (that is the whole point of running two). With remote write, "both receive everything" means declaring one exporter per replica and fanning out in the pipeline:

exporters:
  prometheusremotewrite/prom-0:
    endpoint: http://prometheus-kube-prom-stack-kube-prome-prometheus-0.prometheus-operated:9090/api/v1/write
    tls:
      insecure: true
  prometheusremotewrite/prom-1:
    endpoint: http://prometheus-kube-prom-stack-kube-prome-prometheus-1.prometheus-operated:9090/api/v1/write
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      exporters: [prometheusremotewrite/prom-0, prometheusremotewrite/prom-1]

The per-pod DNS names come from the prometheus-operated headless service the operator already creates — that is what headless services are genuinely useful for here: addressing individual replicas, not fanning out.

Trade-offs worth knowing before you commit to this:

• It doubles egress and doubles the failure surface: an outage on one replica backpressures/retries in its own exporter queue. Give each exporter its own retry_on_failure and remote_write_queue so a single sick replica cannot stall the other.
• The replica count is now baked into your collector config. Scaling the Prometheus StatefulSet means editing the collector.
• Deduplication is on you at query time (Thanos/Grafana datasource dedup, or just accept both).

If this is going to grow, the cleaner shape is to point the collector at something built to ingest remote-write with replication — Mimir or Thanos Receive both do this natively and remove the fan-out from your collector config entirely. Worth considering before you hard-code two endpoints.