prometheusremotewrite

[linusboo]

Hey people!

I'm trying to deploy an OpenTelemetry Collector using the prometheusremotewrite receiver, but I can't seem to get it to start. I'm using the official Helm chart to install the collector. Here’s my YAML configuration:

mode: daemonset

image:
  repository: otel/opentelemetry-collector-contrib
  tag: 0.145.0

networkPolicy:
  enabled: true
service:
  enabled: false

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

config:
  receivers:
    prometheusremotewrite:
      endpoint: 0.0.0.0:19291

  processors:
    batch: {}

  exporters:
    debug:
      verbosity: detailed

  service:
    pipelines:
      metrics:
        receivers: [prometheusremotewrite]
        processors: [batch]
        exporters: [debug]

In addition, I can provide the service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  selector:
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
    protocol: TCP
  - name: otlp-http
    port: 4318
    targetPort: 4318
    protocol: TCP
  - name: prom-remote-write-reciever
    port: 19291
    targetPort: 19291
    protocol: TCP

I can deploy the collector without any warnings, but I don’t see any logs indicating that the prometheusremotewrite receiver has started. Does anyone have suggestions on what might be going wrong?

[s3onghyun]

A couple of Helm-chart specifics are likely biting you here, separate from the receiver itself (which is fine — prometheusremotewritereceiver is alpha: [metrics] but needs no feature gate to start).

1. The chart doesn't open your receiver's port just because the config uses it. The chart exposes container/service ports from its own ports: value, not by scanning your config:. otlp (4317/4318) is in the chart defaults, so it "just works"; 19291 is not, so nothing is listening on it at the Pod level. The service.yaml you wrote by hand is outside the chart, so it points at a targetPort: 19291 the container never opened. Add the port through the chart instead:

ports:
  prom-rw:
    enabled: true
    containerPort: 19291
    servicePort: 19291
    protocol: TCP

(and drop the hand-written Service — let the chart manage it, especially since you also set service.enabled: false, which suppresses the chart's Service entirely).

2. Your config: is deep-merged with the chart's default config, not replacing it. The chart ships a default service.pipelines (with an otlp-based metrics pipeline). Your metrics pipeline overrides that key, so that part is fine, but it's worth running helm template and checking the rendered service.pipelines is exactly what you expect — merge surprises are the most common "it deploys clean but nothing happens" cause with this chart.

3. "No receiver start log" isn't necessarily an error. The receiver logs on startup at info, but if the port never bound (point 1) you also won't see it serving. Quickest way to localize it:

• kubectl exec into the pod and ss -tlnp | grep 19291 — is the collector actually listening?
• turn on the collector's own telemetry (service.telemetry.logs.level: debug) and check for the receiver in the startup component list;
• confirm what's actually running with kubectl get cm <release>-opentelemetry-collector -o yaml — that's the merged config the collector booted with, which is the real source of truth over your values file.

My guess is ss shows nothing on 19291 and the fix is exposing the port via the chart's ports:. If it is listening and data still isn't arriving, then it's the sender side / NetworkPolicy (you have networkPolicy.enabled: true, which will drop ingress to 19291 unless you allow it).